Creating a CentOS mirror site - Revision history

Ppapa at 04:00, 31 October 2011

2011-10-31T04:00:37Z

← Older revision		Revision as of 04:00, 31 October 2011
Line 35:		Line 35:
	You should now have a directory with GPG keys for all the versions and few other directories.		You should now have a directory with GPG keys for all the versions and few other directories.

	5. Use wget to download the version of CentOS that you need, in my case 5.6, while excluding the ISO installer images.		5. Use wget to download the version of CentOS that you need, in my case 6.0, while excluding the ISO installer images.

	wget -r -np -nH --cut-dirs=2\		wget -r -np -nH --cut-dirs=2\
	--include-directories=pub/centos/5.6 \		--include-directories=pub/centos/6.0 \
	--exclude-directories=pub/centos/5.6/isos http://myfavouritemirror.com/pub/centos/		--exclude-directories=pub/centos/3\,pub/centos/4\,pub/centos/5\*,pub/centos/6,pub/centos/6.0/isos \
			http://myfavouritemirror.com/pub/centos/

	The new wget arguments do the following:		The new wget arguments do the following:
Line 48:		Line 49:
	\|}		\|}

	This will obviously download just the 5.6 directory and its contents, while excluding the the isos directory for this version.		This will obviously download just the 6.0 directory and its contents, while excluding the the isos directory for this version.

	You've now created your mirror site!		You've now created your mirror site!
Line 55:		Line 56:

	wget -r -N -np -nH --cut-dirs=2\		wget -r -N -np -nH --cut-dirs=2\
	--include-directories=pub/centos/5.6 \		--include-directories=pub/centos/6.0 \
	--exclude-directories=pub/centos/5.6/isos http://myfavouritemirror.com/pub/centos/		--exclude-directories=pub/centos/3\,pub/centos/4\,pub/centos/5\*,pub/centos/6,pub/centos/6.0/isos \
			http://myfavouritemirror.com/pub/centos/

	where the extra arguement is:		where the extra arguement is:

Ppapa at 03:58, 31 October 2011

2011-10-31T03:58:53Z

← Older revision		Revision as of 03:58, 31 October 2011
Line 1:		Line 1:
	The best way to create a CentOS mirror is to use '''rsync'''. This has been documented [http://wiki.centos.org/HowTos/CreateLocalMirror elsewhere]. However, if you are like me and you're stuck behind a corporate firewall and no one is willing to open port 873 for you, then you can do it all with '''wget''' over port 80. Each mirror site, usually has repositories for multiple versions of CentOS. I only wanted to mirror the repository for CentOS 5.6. Here's how I did it.		The best way to create a CentOS mirror is to use '''rsync'''. This has been documented [http://wiki.centos.org/HowTos/CreateLocalMirror elsewhere]. However, if you are like me and you're stuck behind a corporate firewall and no one is willing to open port 873 for you, then you can do it all with '''wget''' over port 80. Each mirror site, usually has repositories for multiple versions of CentOS. I only wanted to mirror the repository for CentOS 6.0. Here's how I did it.

	1. Set up a server running apache. I'm sure you'll figure it out somewhere else!		1. Set up a server running apache. I'm sure you'll figure it out somewhere else!
Line 15:		Line 15:
	cd /export/htdocs/pub/centos		cd /export/htdocs/pub/centos
	wget -r -np -nH --cut-dirs=2 \		wget -r -np -nH --cut-dirs=2 \
	--exclude-directories=pub/centos/3\,pub/centos/4\,pub/centos/5\* \		--exclude-directories=pub/centos/3\,pub/centos/4\,pub/centos/5\,pub/centos/6\ \
	http://myfavouritemirror.com/pub/centos/		http://myfavouritemirror.com/pub/centos/

Ppapa at 03:55, 31 October 2011

2011-10-31T03:55:25Z

← Older revision		Revision as of 03:55, 31 October 2011
Line 15:		Line 15:
	cd /export/htdocs/pub/centos		cd /export/htdocs/pub/centos
	wget -r -np -nH --cut-dirs=2 \		wget -r -np -nH --cut-dirs=2 \
	--exclude-directories=pub/centos/3.9,pub/centos/4,pub/centos/4.3,pub/centos/4.4,pub/centos/4.5,pub/centos/4.6,pub/centos/4.7,pub/centos/4.8,pub/centos/4.9,pub/centos/5,pub/centos/5.0,pub/centos/5.1,pub/centos/5.2,pub/centos/5.3,pub/centos/5.4,pub/centos/5.5,pub/centos/5.6 \		--exclude-directories=pub/centos/3\,pub/centos/4\,pub/centos/5\* \
	http://myfavouritemirror.com/pub/centos/		http://myfavouritemirror.com/pub/centos/

Ppapa at 03:03, 31 October 2011

2011-10-31T03:03:56Z

← Older revision		Revision as of 03:03, 31 October 2011
Line 14:		Line 14:

	cd /export/htdocs/pub/centos		cd /export/htdocs/pub/centos
	wget -r -np -nH --cut-dirs=2 --exclude-directories=pub/centos/3.9,pub/centos/4,pub/centos/4.3,pub/centos/4.4,pub/centos/4.5,pub/centos/4.6,pub/centos/4.7,pub/centos/4.8,pub/centos/4.9,pub/centos/5,pub/centos/5.0,pub/centos/5.1,pub/centos/5.2,pub/centos/5.3,pub/centos/5.4,pub/centos/5.5,pub/centos/5.6 http://myfavouritemirror.com/pub/centos/		wget -r -np -nH --cut-dirs=2 \
			--exclude-directories=pub/centos/3.9,pub/centos/4,pub/centos/4.3,pub/centos/4.4,pub/centos/4.5,pub/centos/4.6,pub/centos/4.7,pub/centos/4.8,pub/centos/4.9,pub/centos/5,pub/centos/5.0,pub/centos/5.1,pub/centos/5.2,pub/centos/5.3,pub/centos/5.4,pub/centos/5.5,pub/centos/5.6 \
			http://myfavouritemirror.com/pub/centos/

	The wget arguments do the following:		The wget arguments do the following:

Ppapa at 05:58, 13 July 2011

2011-07-13T05:58:18Z

@@ Line 43: / Line 43: @@
 {|
 | '''--include-directories''' || I think this is pretty obvious!
 |-
 |}

Ppapa at 05:54, 13 July 2011

2011-07-13T05:54:02Z

← Older revision		Revision as of 05:54, 13 July 2011
Line 37:		Line 37:
	wget -r -np -nH --cut-dirs=2\		wget -r -np -nH --cut-dirs=2\
	--include-directories=pub/centos/5.6 \		--include-directories=pub/centos/5.6 \
	--exclude-directories=pub/centos/5.6/isos \		--exclude-directories=pub/centos/5.6/isos http://myfavouritemirror.com/pub/centos/
	http://myfavouritemirror.com/pub/centos/

	The new wget arguments do the following:		The new wget arguments do the following:

Ppapa at 05:53, 13 July 2011

2011-07-13T05:53:38Z

@@ Line 28: / Line 28: @@
 |-
 | '''--exclude-directories=.....''' || Comma separated list of directories you don't want to download.
 |-
 |}

Ppapa at 05:47, 13 July 2011

2011-07-13T05:47:08Z

← Older revision		Revision as of 05:47, 13 July 2011
Line 25:		Line 25:
	\| '''-nH''' \|\| Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.		\| '''-nH''' \|\| Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.
	\|-		\|-
	\| '''--cut-dirs=2''' \|\| Along with the -nH argument, wget would create the directories /pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.		\| '''--cut-dirs=2''' \|\| Along with the -nH argument, wget would create the directories pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.
	\|-		\|-
	\| '''--exclude-directories=.....''' \|\| Comma separated list of directories you don't want to download.		\| '''--exclude-directories=.....''' \|\| Comma separated list of directories you don't want to download.
	\|-		\|-
	\|}		\|}

Ppapa at 05:46, 13 July 2011

2011-07-13T05:46:26Z

← Older revision		Revision as of 05:46, 13 July 2011
Line 23:		Line 23:
	\| '''-np''' \|\| Do not ascend to the parent directory when retrieving recursively		\| '''-np''' \|\| Do not ascend to the parent directory when retrieving recursively
	\|-		\|-
	-nH Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.		\| '''-nH''' \|\| Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.
	--cut-dirs=2 Along with the -nH argument, wget would create the directories /pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.		\|-
	--exclude-directories=..... Comma separated list of directories you don't want to download.		\| '''--cut-dirs=2''' \|\| Along with the -nH argument, wget would create the directories /pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.
			\|-
			\| '''--exclude-directories=.....''' \|\| Comma separated list of directories you don't want to download.
			\|-
	\|}		\|}

Ppapa at 05:45, 13 July 2011

2011-07-13T05:45:14Z

← Older revision		Revision as of 05:45, 13 July 2011
Line 21:		Line 21:
	\| '''-r''' \|\| recursively download		\| '''-r''' \|\| recursively download
	\|-		\|-
	\|}		\| '''-np''' \|\| Do not ascend to the parent directory when retrieving recursively
	-np Do not ascend to the parent directory when retrieving recursively		\|-
	-nH Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.		-nH Disable generation of host-prefixed directories. By default, invoking Wget with ‘-r http://myfavouritemirror.com’ will create a structure of directories beginning with myfavouritemirror.com/. This option disables such behavior.
	--cut-dirs=2 Along with the -nH argument, wget would create the directories /pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.		--cut-dirs=2 Along with the -nH argument, wget would create the directories /pub/centos/. This gets rid of them so the data is downloaded directly into the directory you're in.
	--exclude-directories=..... Comma separated list of directories you don't want to download.		--exclude-directories=..... Comma separated list of directories you don't want to download.
			\|}