Docker_Cookbook.pdf

Tableofcontents Preface 1.GettingStartedwithDocker 15 1.1InstallingDockeronUbuntu14.04andCentOS6.5 15 1.2SettingUpaLocalDockerHostUsingVagrant 17 1.3Usingboot2dockertogetaDockerhostonosx 1.4RunningBoot2dockeronWindows8.1Desktop 22 1.5StartingaDockerhostintheCloudUsingdockermachine 24 1.6RunninghelloWorldinDocker 28 1.7RunningaDockerContainerinDetachedmode 1.8Creating,Starting,Stopping,RemovingContainers 31 1.9SharinghostDataWithContainers 1.10SharingDataBetweenContainers 34 1.11CopyingDataToAndFromContainers 1.12ManagingandConfiguringthedockerdaemon 36 1.13RunningaWordpressBlogUsingTwoLinkedContainers 37 1.14BackingupaDatabaseRunninginaContainer 1.15USingSupervisortoRunWordpressinaSingleContainer 41 2.ImageCreationandSharing.........................45 2.1KeepingChangesmadetoaContainerbycommittingtoanimage 45 2.2SavingImagesandContainersasTarFilesforSharing 2.3WritingyourFirstDockerfile 2.4PackagingaFlaskApplicationinsideacontainer 50 2.5VersioninganImagewithTags 53 2.6MigratingFromVagranttoDockerWiththedockerProvider 54 2.7USingPackertoCreateaDockerImage 2.8PublishingyourimagetoDockerhub 2.9RunningaPrivateRegistry 2.10SettingUpanAutomatedBuildonDockerHubforContinous Integration/Deployment 66 2.11SettingupaLocalAutomatedBuildUsingaGitHookandaPrivate Registry 3.Docker 3.1IntroducingDockerContainersNetworking 73 3.2ChoosingaContainerNetworkingStack 3.3ConfiguringtheDockerDaemonIPtablesandIPforwardingsettings 78 3.4LinkingContainersinDocker 80 3.5UsingPipeworktoUnderstandContainerNetworking 80 3.6SettingupaCustomBridgeforDocker 3.7USingovswithDocker 86 3.8BuildingagretunnelBetweenDockerHosts 88 3.9NetworkingContainersonMultipleHostswithDockerNetwork 91 3.10DivingDeeperIntoTheDockerNetworkNamespacesConfiguration 95 3.11RunningContainersonaWeaveNetwork 96 3.12RunningaWeaveNetworkonAwS 97 3.13DeployingflannelOverlayBetweenDockerHosts 99 3.14USinganAmbassadorContainertoExposeServices 101 4.DockerConfigurationanddevelopment......................................103 4.1CompilingYourOwnDockerBinaryFromSource 103 4.2RunningtheDockerTestSuiteforDockerDevelopment 105 4.3ReplacingYourCurrentDockerBinaryWithaNewOne 106 4.4Usingsenter 4.5Introducinglibcontainer 110 4.6Usingnsinit 110 4.7SwitchingExecutionEnvironment 110 4.8AccessingtheDockerDaemonRemotely 110 4.9Exploringthedockerremoteapitoautomatedockertasks 112 4.10SecuringtheDockerDeamonforRemoteaccess 114 4.11USingdocker-pytoAccesstheDockerDaemonRemotely 116 4.12Usingdocker-pySecurely 118 5.Kubernetes 121 5.1UnderstandingKubernetesArchitecture 123 5.2NetworkingPodsforContainerConnectivity 126 5.3UsingLabelsforContainerPlacementandApplicationManagement 129 5.4CreatingaMulti-nodeKubernetesClusterWithVagrant 129 5.5StartingContainersonaKubernetesClusterwithPods 132 ivTableofContents 5.6TakingAdvantageofLabelsForQueryingKubernetesObjects 134 5.7UsingaReplicationControllertoManagetheNumberofReplicasofa Pod 135 5.8Runningmultiplecontainersinapod 137 5.9UsingServiceProxiesForDynamicLinkingofContainers 140 5.10DefiningVolumesinPods 142 5.11CreatingaSingleNodeKubernetesClusterUsingDockerCompose 143 5.12CompilingKubernetestoCreateYourOwnRelease 146 5.13StartingKubernetesComponentswithhyperkubeBinary 149 5.14ExploringtheKubernetesAPI 150 5.15RunningtheKubernetesdashboard 154 5.16SwitchingtoaNewAPiversion 156 5.17ConfiguringAuthenticationtoaKubernetesCluster 158 5.18ConfiguringtheKubernetesclienttoAccessremoteclusters 159 6.JustEnoughOperatingSystemforDocker...................161 6.1DiscoveringtheCoresLinuxDistributionwithvagrant 161 6.2StartingaContaineronCoreoSviaCloud-init 164 6.3StartingaCoreOSClusterviaVagranttoRunContainersonMultiple Hosts 166 6.4UsingFleettoStartContainersonaCoreOSCluster 169 6.5DeployingaflanneloverlaybetweenCoreosInstances 171 6.6RunningDockerContainersonRancherOS 174 6.7UsingProjectAtomictorunDockerContainers 175 6.8StartingandAtomicInstanceonAWStouseDocker 176 6.9RunningDockeronUbuntuCoreSnappyinaSnap 6.10StartinganUbuntuCoreSnappyInstanceonAWSEC2 179 7.ThedockerEcosystem:Tools. 185 7.1UsingDockercomposetoCreateaWordpressSite 185 7.2UsingDockercomposetotestApacheMesosandMarathononDocker188 7.3LookingatDockerComposeasaReplacementtoFig 190 7.4StartingContainersonaClusterwithDockerSwarm 193 7.5UsingDockermachinetoCreateaSwarmClusterAcrossCloudProviders196 7.6ManagingContainersthroughDockerU 198 7.7OrchestratingContainerswithAnsibleDockerModule 200 7.8USingClocker 204 7.9UsingDeis 204 7.10UsingRanchertoManageContainersonaClusterofDockerHosts 204 7.11RunningContainersviaapacheMesosandmarathon 208 7.12UsingtheMesosDockerContainerizeronaMesosCluster 213 7.13DiscoveringDockerServiceswithRegistrator 215 Tableofcontents 8.Dockerinthecloud 219 8.1AccessingPublicCloudstoRunDocker 220 8.2StartingaDockerHostonAWSEC2 223 8.3StartingaDockerHostongooglegce 226 8.4StartingaDockerHostonMicrosoftAzure 229 8.5StartingadockerHostonazurewithdockermachine 231 8.6RunningCloudProvidersCLIinDockerContainers 233 8.7UsinggoogleContainerregistrytoStoreyourdockerimages 235 8.8UsingdockerinGCEGoogle-ContainerInstances 237 8.9StartingaDockerHostonAwSUsingDockerMachine 240 8.10UsingKubernetesintheCloudviaGoogleContainerEngine 242 8.11ManagingGoogleContainerEngineResourcesUsingkubecfg 244 8.12GettingSetuptoUsetheEC2ContainerService 246 8.13CreatingaECSCluster 249 8.14StartingDockerContainersonaECSCluster 252 8.15StartinganApplicationintheCloudUsingDockerSupportinAWS Beanstalk 256 8.16UsingAWSElasticContainerServiceasaBeanstalkEnvironment 260 9.Monitoringcontainers...........................261 9.1GettingDetailedInformationAboutaContainerWithdockerinspect261 9.2ObtainingUsageStatisticsofaRunningContainer 263 9.3ListeningtoDockerEventsonYourDockerHosts 264 9.4GettingThelogsofaContainerWithdockerlogs 266 9.5UsingLogspouttoCollectContainerLogs 267 9.6ManaginglogspoutRoutestoStoreContainerLogs 269 9.7UsingelasticsearchandKibanatoStoreandVisualizeContainerlogs271 9.8UsingCollectdtoVisualizeContainerMetrics 272 9.9AccessingContainerLogsThroughMountedVolumes 278 9.10UsingcAdvisortoMonitorResourceUsageinContainers 278 9.11MonitoringContainerMetricsWithInfluxDB,GrafanaandcAdvisor280 9.12GainingVisibilityIntoYourContainersLayoutwithWeavescope 281 9.13MonitoringaKubernetesclusterwithheapster 283 Tableofcontents Preface Thisbookisnotfinished,ithasnotgonethroughtechnicalreview norhasitbeeneditedforgrammar,punctuation,typosetc.Youare OReilly,pleaseconsideritapreviewofadraftandbekindinyour reviews.Feelfreetosendmeanysuggestionsorcommentsto how2dock@gmail.com.YoucanalsofileareviewintheOReilly portal.Happyreading WhyIWroteThisBook IhavebeenworkingonCloudsattheIaaslayerforovertenyears.WithAmazon AWS,GoogleGCeandMicrosoftAzurenowprovidinglargescaleCloudservicesfor severalyears,itisfairtosaythatgettingaccesstoaserverhasneverbeenthateasy andthatquick.Therealvaluetomehasbeentheavailabilityofanapitoaccessthese services.Wecannowprogramtocreateaninfrastructureandprogramtodeployan application.Theseprogrammablelayershelpusreachahigherlevelofautomation, whichforabusinesstranslatesinfastertimetomarket,moreinnovationandbetter userservice However,applicationpackaging,configuration,compositionofservicesinadis tributedenvironmenthasnotprogressedmuchdespitealotofworkinconfiguration managementandorchestration.Deployingandrunningadistributedapplicationat scaleandinafaulttolerantmannerisstillhard IwasnotcrazyaboutDockeruntilItrieditandunderstoodwhatitbringstothe table.Dockerprimarilybringsanewuserexperiencetolinuxcontainers.Itisnot aboutfullvirtualizationversuscontainers,itisabouttheeaseofpackagingandrun ninganapplication.OnceyoustartusingDockerandenjoythisnewexperience,the sideeffectisthatyouwillalsostartthinkingautomaticallyaboutcompositionand Clustering Containershelpusthinkmoreintermsoffunctionalisolationwhichinturnforcesus todecomposeourapplicationsbeforestitchingitbacktogetherforadistributed world.Yes,IhavedrunktheKool-Aidbuthopefullythisbookwillshowyouwhy HowThisbookIsorganized Thiscookbookiscurrentlymadeoftenchapters.Eachchapteriscomposedofrecipes writteninthestandardOreillyrecipesformat(i.eProblem,Solution,Discussion Youcanreaditfronttobackorpickupaspecificchapterrecipe.Eachrecipeismade tobeindependentoftheothersbutwhenconceptsneededinarecipeareneeded appropriatereferencesareprovided TheChapter1chaptergoesthroughseveralDockerinstallationscenariosinclud ingDockermachine.ItthenpresentsthebasicDockercommandstomanagecon tainers,mountdatavolumes,linkcontainersandsoon.Attheendofthischapter youshouldhaveaworkingdockerhostandyoushouldhavestartedmultiple containersaswellasunderstoodthelifecycleofcontainers InChapter2weintroducetheDockerfile,DockerHubandshowhowto build/tag/commitanimage.WealsoshowhowtorunyourownDockerregistry andsetupautomatedbuilds.Attheendofthischapteryouwillknowhowtocre ateDockerimagesandsharethemprivatelyorpubliclyandhavesomebasic foundationtobuildcontinuousdeliverypipelines CurrentlytheChapter3chapteronlycontainsstubsofplannedrecipes.Youwill findinformationsuchaslinkingcontainers,usinganambassadorcontainerto exposeservicesfromdifferenthosts,configuringacustombridgeforuseinyour Dockerhost.Youwillalsolearnaboutmoreadvancednetworkingsetupsand tools,suchasWeave,FlannelandSocketplaneVXLANoverlaysEmptychapter rightnow TheChapter4chaptergoesthroughsomeconfigurationoftheDockerdaemon, especiallysecuritysettingsandaccesstotheDockerAPiremotely.Italsocoversa fewbasicproblems,likecompilingDockerfromsource,runningitstestsuiteand usinganewDockerbinary.afewrecipesaremeanttogainabetterinsighton linuxnamespacesanditsuseincontainers Chapter5introducesthenewcontainermanagementplatformfromgoogle Kubernetesprovidesawaytodeploymulticontainerapplicationsonadis tributedcluster.Inadditionitprovidesanautomatedwaytoexposeservicesand createreplicasofcontainers.Weshowhowtodeploykubernetesonyourown infrastructure,startingwithalocalVagrantclusterandsubsequentlyonasetof machinesstartedintheCloud.WethenpresentthekeyaspectsofKubernetes pods,servicesandreplicationcontrollers.Emptychapterrightnow. IPreface IntheChapter6chapterwecoverthreenewlinuxdistributionsthatarecustom izedtoruncontainers:CoreOS,ProjectAtomicandUbuntucore.Thesenewdis- tributionsprovidejustenoughoperatingsystemtorunandorchestratedocker containers.Recipescoverinstallationandaccesstomachinesthatusethesedis tributions.Wealsointroducetoolsthatareusedwiththesedistributionstoease containerorchestration(e.getcd,fleet,systemd OneofDocker'sstrengthisitsboomingecosystem.InChapter7weintroduce severaltoolsthathavebeencreatedoverthelast18monthsandthatleverage Dockertoeaseapplicationdeployment,continuousintegration,servicediscovery andorchestration.Asanexample,youwillfindrecipesaboutFig,Docker Swarm,Flynnandapacheclocker TheDockerdaemoncanbeinstalledonadeveloperlocalmachine,however, withCloudcomputingprovidingeasyaccesstoon-demandserversitisfairto saythatalotofcontainerbasedapplicationswillbedeployedintheCloud.In Chapter8wepresentafewrecipestoshowhowtoaccessaDockerhostonama zonAWS,GoogleGCEandMicrosoftAzure.Wealsointroducetwonewcloud servicesthatuseDocker:TheawSElasticContainerService(ECS)andthegoo gleContainerEngine.Thischapteriscurrentlystubbedout.Emptychapterright now TheChapter9chaptersaimstoaddresssomeconcernsaboutapplicationmoni- toringwhenusingcontainers.Monitoringandvisibilityoftheinfrastructureand theapplicationhasbeenahugefocusintheDevOpscommunity.AsDocker becomesmorepervasiveasadevelopmentandoperationalmechanism,lessons learnedneedtobeappliedtocontainerbasedapplications.Thischapteriscur rentlystubbedoutEmptychapterrightnow Finally,inthe?chapterwepresentendtoendapplicationdeploymentscenar iosonbothsinglehostandclusters.Whilesomebasicapplicationdeployments arepresentedinearlierchaptersTherecipespresentedhereaimtobeclosertoa productiondeploymentsetup.Thisisamoreindepthchapterthatputsthe readeronthepathtowardsdesigningmorecomplexmicroservices.Thischapter iscurrentlystubbedout.Emptychapterrightnow. Thebookstructuremaystillchangeandtheorderofthechapters maybemodified Finally,??summarizesthebookandprovidessometipsforfurtherreadingand investigation Preface