distcc (Русский)
Distcc это программа, предназначенная для распределения задач компиляции между компьютерами в сети на таких языках как C, C++, Objective C or Objective C++. Приведет к тем же результатам, что и локальная сборка, простота в установке и использовании и как правило, гораздо быстрее, чем локальная компиляция. Круто то что можно использовать его вместе с родными утилитами для компиляции Arch, такими как makepkg.
Contents
Условия
- master
- Мастер-это компьютер, на котором выполняется компиляция.
- slaves
- Подчиненный(е) принимает запросы на компиляцию, отправленные мастером.
Приступая к работе
Установить пакет distcc на всех компьютерах в кластере:
Для других дистрибутивов, или даже операционных систем, включая Windows используйте через cygwin, обратитесь к distcc docs.
Конфигурация
Ведомый (Slaves)
Конфигурация для slaves хранится в /etc/conf.d/distccd
. Параметры командной строки указаны в distcc(1). Как минимум, настроить разрешенные диапазонов адресов в CIDR формате:
DISTCC_ARGS="--allow 192.168.0.0/24"
Хороший инструмент для преобразования диапазонов адресов CIDR можно найти здесь: CIDR Utility Tool.
Запустить distccd.service
на всех участвующих slave. Для запуска distccd.service
при загрузке, enable на каждой участвующей машине.
Мастер (Master)
Для использования с makepkg
Редактировать /etc/makepkg.conf
в следующих трех разделах:
- BUILDENV has distcc unbanged i.e. without exclamation point.
- Uncomment the DISTCC_HOSTS line and add the IP addresses of the slaves then a slash and the number of threads they are to use. The subsequent IP address/threads should be separated by a white space. This list is ordered from most powerful to least powerful (processing power).
- Adjust the MAKEFLAGS variable to correspond to the number of sum of the number of individual values specified for the max threads per server. In the example below, this is 5+3+3=11. If users specify more than this sum, the extra theoretical thread(s) will be blocked by distcc and appear as such in monitoring utils such as distccmon-text described below.
Пример с использованием соответствующих строк:
BUILDENV=(distcc fakeroot color !ccache check !sign) MAKEFLAGS="-j11" DISTCC_HOSTS="192.168.0.2/5 192.168.0.3/3 192.168.0.4/3"
Если пользователи хотят использовать distcc через SSH, добавьте символ "@" перед IP-адресом в этом разделе. Если аутентификация на основе ключей в системах не настроена, установите переменную DISTCC_SSH, чтобы игнорировать проверку для аутентифицированных хостов, то есть DISTCC_SSH="ssh -i"
For use without makepkg
The minimal configuration for distcc on the master includes the setting of the available slaves. This can either be done by setting the addresses in the environment variable DISTCC_HOSTS
or in either of the configuration files $DISTCC_HOSTS
, $DISTCC_DIR/hosts
, ~/.distcc/hosts
or /etc/distcc/hosts
.
Example for setting the slave address using DISTCC_HOSTS
:
$ export DISTCC_HOSTS="192.168.0.3,lzo,cpp 192.168.0.4,lzo,cpp"
Example for setting the slave addresses in the hosts configuration file:
~/.distcc/hosts
192.168.0.3,lzo,cpp 192.168.0.4,lzo,cpp
Instead of explicitly listing the server addresses one can also use the avahi zeroconf mode. To use this mode +zeroconf
must be in place instead of the server addresses and the distcc daemons on the slaves have to be started using the --zeroconf
option. Note that this option does not support the pump mode!
The examples add the following options to the address:
-
lzo
: Enables LZO compression for this TCP or SSH host (slave). -
cpp
: Enables distcc-pump mode for this host (slave). Note: the build command must be wrapped in the pump script in order to start the include server.
A description for the pump mode can be found here: HOW DISTCC-PUMP MODE WORKS and distcc's pump mode: A New Design for Distributed C/C++ Compilation
To use distcc-pump mode for a slave, users must start the compilation using the pump script otherwise the compilation will fail.
Compile
With makepkg
Compile via makepkg as normal.
Without makepkg
To compile a source file using the distcc pump mode, use the following command:
$ pump distcc g++ -c hello_world.cpp
In this case the pump script will execute distcc which in turn calls g++ with "-c hello_world.cpp" as parameter.
To compile a Makefile project, first find out which variables are set by the compiler. For example in gzip-1.6, one can find the following line in the Makefile: CC = gcc -std=gnu99
. Normally the variables are called CC
for C projects and CXX
for C++ projects. To compile the project using distcc it would look like this:
$ wget ftp://ftp.gnu.org/pub/gnu/gzip/gzip-1.6.tar.xz $ tar xf gzip-1.6.tar.xz $ cd gzip-1.6 $ ./configure $ pump make -j2 CC="distcc gcc -std=gnu99"
This example would compile gzip using distcc's pump mode with two compile threads. For the correct -j
setting have a look at What -j level to use?
Monitoring progress
Progress can be monitored via several methods.
- distccmon-text
- tailing log file
Invoke distccmon-text to check on compilation status:
$ distccmon-text 29291 Preprocess probe_64.c 192.168.0.2[0] 30954 Compile apic_noop.c 192.168.0.2[0] 30932 Preprocess kfifo.c 192.168.0.2[0] 30919 Compile blk-core.c 192.168.0.2[1] 30969 Compile i915_gem_debug.c 192.168.0.2[3] 30444 Compile block_dev.c 192.168.0.3[1] 30904 Compile compat.c 192.168.0.3[2] 30891 Compile hugetlb.c 192.168.0.3[3] 30458 Compile catalog.c 192.168.0.4[0] 30496 Compile ulpqueue.c 192.168.0.4[2] 30506 Compile alloc.c 192.168.0.4[0]
One can have this program run continuously by using watch or by appending a space followed by integer to the command which corresponds to the number of sec to wait for a repeat query:
$ watch distccmon-text
or
$ distccmon-text 2
One can also simply tail systemd journal log
on daemon:
# journalctl -f -u distccd
"Cross Compiling" with distcc
X86
There are currently two methods from which to select to have the ability of distcc distribution of tasks over a cluster building i686 packages from a native x86_64 environment. Neither is ideal, but to date, there are the only two methods documented on the wiki.
An ideal setup is one that uses the unmodified ARCH packages for distccd running only once one each node regardless of building from the native environment or from within a chroot AND one that works with makepkg. Again, this Utopian setup is not currently known.
A discussion thread has been started on the topic; feel free to contribute.
Метод Chroot (предпочтительно)
Assuming the user has a 32-bit chroot setup and configured on each node of the distcc cluster, the strategy is to have two separate instances of distccd running on different ports on each node -- one runs in the native x86_64 environment and the other in the x86 chroot on a modified port. Start makepkg via a schroot command[broken link: invalid section] invoking makepkg.
Добавить номера портов в DISTCC_HOSTS для i686 chroot
Добавьте номер порта, определенный ранее (3692), к каждому из хостов в /opt/arch32/etc/makepkg.conf
следующим образом:
DISTCC_HOSTS="192.168.1.101/5:3692 192.168.1.102/5:3692 192.168.1.103/3:3692"
Вызов makepkg из родной среды
Setup schroot[broken link: invalid section] on the native x86_64 environment. Invoke makepkg to build an i686 package from the native x86_64 environment, simply by:
$ schroot -p -- makepkg -src
Multilib GCC method (not recommended)
See Makepkg#Build 32-bit packages on a 64-bit system.
Other architectures
Arch Linux ARM
Разработчики настоятельно рекомендуют использовать официальный проект toolchains, который должен быть установлен на подчиненном (slave) компьютере(ах) x86_64. Вместо того, чтобы вручную управлять ими, AUR предоставляет все четыре набора инструментов, а также простые модули обслуживания systemd:
Настройка на ведомом компьютере, содержащем набор инструментов, идентична #Slaves[broken link: invalid section] за исключением того, что имя файла конфигурации и служебного файла systemd совпадает с именем соответствующего пакета. Например, для armv7h файл конфигурации - /etc/conf.d/distccd-armv7h
, а системный сервисный модуль - distccd-armv7h.service
.
Обратите внимание, что каждая из цепочек инструментов работает на своем уникальном порту, что позволяет им если это необходимо сосуществовать на подчиненном компьютере. Обязательно разрешите трафик на порт, на котором работает distcc, см. Category:Firewalls и distcc(1).
Target architecture | Distcc Port |
---|---|
armv5 | 3633 |
armv6h | 3634 |
armv7h | 3635 |
armv8h/aarch64 | 3636 |
Master
Настройка мастера идентична #Master за исключением того, что необходимо изменить следующие два файла, чтобы определить теперь нестандартный порт, который предполагается использовать для подчиненных. Обратитесь к таблице выше, если используете пакет AUR.
-
/etc/conf.d/distcc
: пример на машине armv7h:DISTCC_ARGS="--allow 127.0.0.1 --allow 192.168.10.0/24 --port 3635
-
/etc/makepkg.conf
: пример на машине armv7h:DISTCC_HOSTS="192.168.10.2/5:3635 192.168.10.3/5:3635"
Additional toolchains
- EmbToolkit: Tool for creating cross compilation tool chain; supports ARM and MIPS architectures; supports building of an LLVM based tool chain
- crosstool-ng: Similar to EmbToolkit; supports more architectures (see website for more information)
- Linaro: Provides tool chains for ARM development
The EmbToolkit
provides a nice graphical configuration menu (make xconfig
) for configuring the tool chain.
Troubleshooting
Journalctl
Use journalctl
to find out what was going wrong:
$ journalctl $(which distccd) -e --since "5 min ago"
code 110
Make sure that the tool chain works for the user account under which the distcc daemon process gets started (default is nobody). The following will test if the tool chain works for user nobody. In /etc/passwd
change the login for the nobody user to the following:
$ cat /etc/passwd
... nobody:x:99:99:nobody:/:/bin/bash ...
Then cd into the directory containing the cross compiler binaries and try to execute the compiler:
# su nobody $ ./gcc --version bash: ./gcc: Permission denied
Users experiencing this error should make sure that groups permissions as described in #Other architectures are correctly in setup.
Make sure to change back /etc/passwd
to its original state after these modifications.
Alternatively, use sudo without changing the shell in /etc/passwd.
# sudo -u nobody gcc --version
Adjust log level
By default, distcc will log to /var/log/messages.log
as it goes along. One trick (actually recommended in the distccd manpage) is to log to an alternative file directly. Again, one can locate this in RAM via /tmp. Another trick is to lower to log level of minimum severity of error that will be included in the log file. Useful if only wanting to see error messages rather than an entry for each connection. LEVEL can be any of the standard syslog levels, and in particular critical, error, warning, notice, info, or debug.
Either call distcc with the arguments mentioned here on the master or appended it to DISTCC_ARGS in /etc/conf.d/distccd
on the slaves:
DISTCC_ARGS="--allow 192.168.0.0/24 --log-level error --log-file /tmp/distccd.log"
Limit HDD/SSD usage by relocating $HOME/.distcc
By default, distcc creates $HOME/.distcc
which stores transient relevant info as it serves up work for nodes to compile. Create a directory named .distcc in RAM such as /tmp and soft link to it in $HOME. This will avoid needless HDD read/writes and is particularly important for SSDs.
$ mv $HOME/.distcc /tmp $ ln -s /tmp/.distcc $HOME/.distcc
Use systemd to re-create this directory on a reboot (the soft link will remain until it is manually removed like any other file):
Create the following tmpfile.
/etc/tmpfiles.d/tmpfs-create.conf
d /tmp/.distcc 0755 <username> users -