From 7b0202658ab57fa32d9e9e659d925bac4773bb0e Mon Sep 17 00:00:00 2001 From: "github-actions[bot]" Date: Mon, 1 Dec 2025 00:05:25 +0000 Subject: [PATCH] docs: auto-sync wiki from main repository Updated: 2025-12-01 00:05:25 UTC Source commit: 2168edd3b676ccde0b1dade2a352a20f07e0d369 Triggered by: push --- GitHub-Actions.md | 529 ++++++++++++++++++++++++ Home.md | 224 +++++++++- Image-RaspiVirt-Incus-Docker.md | 708 ++++++++++++++++++++++++++++++++ Image-RaspiVirt-Incus.md | 593 ++++++++++++++++++++++++++ _Sidebar.md | 25 ++ 5 files changed, 2078 insertions(+), 1 deletion(-) create mode 100644 GitHub-Actions.md create mode 100644 Image-RaspiVirt-Incus-Docker.md create mode 100644 Image-RaspiVirt-Incus.md create mode 100644 _Sidebar.md diff --git a/GitHub-Actions.md b/GitHub-Actions.md new file mode 100644 index 0000000..1ba59ec --- /dev/null +++ b/GitHub-Actions.md @@ -0,0 +1,529 @@ +# GitHub Actions - Automated Build System + +This page documents the automated build system powered by GitHub Actions. The workflow automatically builds all Raspberry Pi images, compresses them, and creates GitHub releases with ready-to-flash `.img.xz` files. + +## Overview + +The GitHub Actions workflow (`.github/workflows/build-images.yml`) provides a complete CI/CD pipeline for building Raspberry Pi images. It automatically: + +1. Detects all available images in the `images/` directory +2. Downloads base images (RaspiOS + Debian) +3. Executes setup scripts in QEMU ARM64 +4. Creates hybrid images +5. Compresses images with PiShrink +6. Creates GitHub releases with downloadable assets + +## Workflow Triggers + +The workflow runs automatically on: + +### 1. Push Events +- **Trigger**: Any push to any branch +- **Branches**: `**` (all branches) +- **Purpose**: Immediate build on code changes +- **Release Type**: + - `main` branch → Stable release + - Other branches → Pre-release + +### 2. Scheduled Builds +- **Trigger**: Daily at 2:00 AM UTC +- **Schedule**: `cron: '0 2 * * *'` +- **Purpose**: Fetch latest Debian and RaspiOS base images +- **Benefit**: Images stay up-to-date with security patches and new features + +### 3. Manual Dispatch +- **Trigger**: Manual workflow run via GitHub Actions UI +- **Purpose**: Build on-demand without pushing code +- **Usage**: Go to Actions tab → Select workflow → "Run workflow" + +## Build Architecture + +The workflow uses a **4-stage parallel build system** with artifact caching to optimize build time and resource usage. + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Job 1: detect-images │ +│ └─ Scans images/ directory and outputs image list │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Job 2: stage1-download (Parallel per image) │ +│ ├─ Downloads RaspiOS and Debian base images │ +│ ├─ Creates setup.iso from setup.sh + setupfiles/ │ +│ └─ Uploads artifacts: *.img, *.raw, seed.img, setup.iso │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Job 3: stage2-qemu (Parallel per image) │ +│ ├─ Downloads artifacts from stage1 │ +│ ├─ Launches QEMU ARM64 with Debian + setup.iso │ +│ ├─ Executes setup.sh in native ARM64 environment │ +│ ├─ Installs RaspiOS kernel/firmware via APT │ +│ └─ Uploads configured Debian image │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Job 4: stage3-build (Parallel per image) │ +│ ├─ Downloads RaspiOS from stage1 │ +│ ├─ Downloads configured Debian from stage2 │ +│ ├─ Runs merge-debian-raspios.sh │ +│ └─ Uploads hybrid image (rpi-*.img) │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Job 5: stage4-compress (Parallel per image) │ +│ ├─ Downloads hybrid image from stage3 │ +│ ├─ Compresses with PiShrink + xz │ +│ └─ Uploads final image (rpi-*.img.xz) │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Job 6: create-release │ +│ ├─ Downloads all final images (rpi-*.img.xz) │ +│ ├─ Generates release name and tag │ +│ ├─ Creates GitHub release with build info │ +│ └─ Uploads all .img.xz files as release assets │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Detailed Job Breakdown + +### Job 1: detect-images + +**Purpose**: Dynamic image detection + +**Steps**: +1. Checkout repository +2. Scan `images/` directory +3. Extract directory names (basename) +4. Convert to JSON array for matrix strategy +5. Output image list to GitHub Actions outputs + +**Output**: JSON array like `["raspivirt-incus", "raspivirt-incus+docker"]` + +**Code**: +```bash +images=$(ls -d images/*/ | xargs -n 1 basename | jq -R -s -c 'split("\n")[:-1]') +``` + +### Job 2: stage1-download + +**Purpose**: Download and prepare base images + +**Strategy**: Matrix parallelization (one job per image) + +**Steps**: +1. **Free disk space**: Remove unused packages to prevent disk full errors + - Removes: .NET, Android SDK, GHC, CodeQL, Docker images + - Frees ~40GB of space +2. **Install dependencies**: `wget`, `genisoimage`, `xz-utils` +3. **Run autobuild stage 1**: `./bin/autobuild --image --stage 1` + - Downloads RaspiOS image (if missing) + - Downloads Debian image (if missing) + - Creates `setup.iso` from `setup.sh` + `setupfiles/` + - Regenerates cloud-init `seed.img` +4. **Upload artifacts**: + - RaspiOS image (`*.img`) + - Debian image (`*.raw`) + - Cloud-init seed (`seed.img`) + - Setup ISO (`setup.iso`) + - Retention: 1 day + +**Artifact Size**: ~2-4GB per image + +### Job 3: stage2-qemu + +**Purpose**: Execute setup script in QEMU ARM64 + +**Strategy**: Matrix parallelization (one job per image) + +**Timeout**: 60 minutes (safety limit for long-running setups) + +**Steps**: +1. **Install QEMU dependencies**: + - `qemu-system-aarch64`: ARM64 system emulator + - `qemu-utils`: Image utilities + - `qemu-efi-aarch64`: UEFI firmware for ARM64 +2. **Download artifacts from stage1**: + - RaspiOS image (needed for reference) + - Debian image (modified during setup) + - Cloud-init seed (first-boot configuration) + - Setup ISO (contains setup.sh + files) +3. **Run autobuild stage 2**: `./bin/autobuild --image --stage 2` + - Creates working copy of Debian image + - Launches QEMU ARM64 with: + - Debian image as main disk + - Cloud-init seed for user configuration + - Setup ISO for package installation + - QEMU executes setup.sh in native ARM64: + - System update (`apt update && apt upgrade`) + - Essential packages (curl, wget, sudo, SSH, etc.) + - RaspiOS repository configuration + APT pinning + - RaspiOS kernel/firmware installation + - Image-specific software (Incus, Docker, etc.) + - Service configuration + - Waits for automatic poweroff signal +4. **Upload configured Debian image**: + - Debian image with RaspiOS kernel + custom software + - Retention: 1 day + - Compression disabled (speed optimization) + +**Artifact Size**: ~3-6GB per image + +**Key Feature**: Native ARM64 execution ensures compatibility and proper package installation + +### Job 4: stage3-build + +**Purpose**: Create hybrid Raspberry Pi image + +**Strategy**: Matrix parallelization (one job per image) + +**Steps**: +1. **Install build dependencies**: + - `parted`: Partition management + - `e2fsprogs`: ext4 filesystem tools + - `dosfstools`: FAT32 filesystem tools + - `rsync`: Efficient file copying + - `qemu-utils`: Image conversion +2. **Enable loop devices**: Load kernel module for disk image mounting +3. **Download artifacts**: + - RaspiOS image from stage1 (boot partition source) + - Configured Debian image from stage2 (rootfs source) +4. **Run autobuild stage 3**: `./bin/autobuild --image --stage 3` + - Executes `merge-debian-raspios.sh`: + - Creates output image based on RaspiOS + - Resizes to configured IMAGE_SIZE + - Mounts both images via loop devices + - Deletes RaspiOS root partition content + - Copies Debian rootfs (with RaspiOS kernel pre-installed) + - Restores RaspiOS `/etc/fstab` + - Creates `/boot/firmware` mount point + - Unmounts and cleans up +5. **Upload hybrid image**: + - Final bootable image (`rpi-*.img`) + - Retention: 1 day + - Compression disabled (PiShrink does this in stage 4) + +**Artifact Size**: Matches IMAGE_SIZE (e.g., 6GB for raspivirt-incus) + +**Result**: Bootable Raspberry Pi image ready for compression + +### Job 5: stage4-compress + +**Purpose**: Shrink and compress final images + +**Strategy**: Matrix parallelization (one job per image) + +**Steps**: +1. **Install PiShrink**: + - Downloads from official repository + - Moves to `/usr/local/bin/` +2. **Download hybrid image from stage3** +3. **Run autobuild stage 4**: Compresses with PiShrink + - Executes: `sudo pishrink.sh -aZ .img` + - `-a`: Aggressive compression + - `-Z`: Parallel xz compression + - Process: + - Shrinks filesystem to minimum size + - Removes unused space from partition + - Truncates image file + - Compresses to `.img.xz` +4. **Upload final image**: + - Compressed image (`rpi-*.img.xz`) + - Retention: 7 days + - Typical compression: 6GB → 1-2GB + +**Artifact Size**: ~1-2GB per image (compressed) + +**Compression Ratio**: Typically 60-80% size reduction + +### Job 6: create-release + +**Purpose**: Create GitHub release with all final images + +**Permissions**: `contents: write` (required for release creation) + +**Steps**: +1. **Download all compressed images**: + - Pattern: `final-image-*` + - Merges all image artifacts +2. **List built images**: Display file sizes for verification +3. **Generate release information**: + - **Release tag**: `v-` (e.g., `v2025-12-01-1430`) + - **Release name**: + - Main branch: `Release ` + - Other branches: `Pre-release ()` + - **Pre-release flag**: + - Main branch: `false` (stable release) + - Other branches: `true` (experimental release) +4. **Create release body**: Markdown with: + - Build date + - Flash instructions + - Build information (branch, commit, workflow link) + - Documentation links +5. **Create GitHub release**: + - Uses `gh release create` CLI + - Uploads all `.img.xz` files as assets + - Sets pre-release flag based on branch +6. **Cleanup**: Remove large files to free disk space + +**Release Example**: +``` +Tag: v2025-12-01-1430 +Name: Release 2025-12-01 +Assets: + - rpi-raspivirt-incus.img.xz (1.2 GB) + - rpi-raspivirt-incus+docker.img.xz (1.5 GB) +``` + +## Artifact Management + +### Artifact Flow + +``` +stage1 → stage2: Base images (RaspiOS, Debian, seed.img, setup.iso) +stage2 → stage3: Configured Debian image +stage3 → stage4: Hybrid image (rpi-*.img) +stage4 → release: Final compressed image (rpi-*.img.xz) +``` + +### Retention Policy + +- **Stage 1-4 artifacts**: 1 day (temporary build artifacts) +- **Final compressed images**: 7 days (backup before release) +- **Release assets**: Permanent (until manually deleted) + +### Compression Strategy + +- **Stage 1-3**: No compression (speed optimization) +- **Stage 4**: Full compression with PiShrink + xz +- **Rationale**: Intermediate artifacts are deleted quickly; only final images need compression + +## Optimization Strategies + +### 1. Parallel Execution +- All images build simultaneously (matrix strategy) +- Total build time = slowest image (not sum of all images) + +### 2. Stage Separation +- Each stage uploads artifacts for the next stage +- Failed stages don't rebuild earlier stages +- Easy to retry individual stages + +### 3. Artifact Caching +- Reuses artifacts across jobs +- Avoids redundant downloads and builds +- Reduces total workflow time + +### 4. Disk Space Management +- Removes unused software before builds +- Cleans up artifacts after each stage +- Prevents out-of-disk errors on GitHub runners + +### 5. Fail-Fast Disabled +- `fail-fast: false` in stage2 (QEMU) +- One failing image doesn't stop others +- Maximizes successful builds + +## Environment Variables + +### Global Environment +```yaml +env: + DEBIAN_FRONTEND: noninteractive +``` +- Prevents interactive prompts during apt operations +- Required for unattended package installation + +### Job-Specific Variables +- Passed via autobuild script stages +- Configured in `images//config.sh`: + - `OUTPUT_IMAGE`: Final filename + - `IMAGE_SIZE`: Target image size + - `QEMU_RAM`: RAM for QEMU + - `QEMU_CPUS`: CPU cores for QEMU + +## Release Versioning + +### Version Format +- **Pattern**: `vYYYY-MM-DD-HHMM` +- **Example**: `v2025-12-01-1430` (December 1, 2025 at 14:30) +- **Uniqueness**: Timestamp ensures unique tags + +### Release Types + +#### Stable Release (main branch) +- **Name**: `Release YYYY-MM-DD` +- **Flag**: `prerelease: false` +- **Purpose**: Production-ready images +- **Visibility**: Featured on repository homepage + +#### Pre-Release (other branches) +- **Name**: `Pre-release YYYY-MM-DD (branch-name)` +- **Flag**: `prerelease: true` +- **Purpose**: Testing and development +- **Visibility**: Listed but marked as pre-release + +## Build Duration + +Typical build times on GitHub runners (ubuntu-latest): + +- **Stage 1 (Download)**: 3-5 minutes per image +- **Stage 2 (QEMU)**: 15-30 minutes per image (most time-consuming) +- **Stage 3 (Build)**: 5-10 minutes per image +- **Stage 4 (Compress)**: 3-5 minutes per image +- **Stage 5 (Release)**: 1-2 minutes + +**Total Time**: ~30-50 minutes for all images (parallel execution) + +## Monitoring Builds + +### Via GitHub UI +1. Go to repository **Actions** tab +2. Select workflow run +3. View job progress and logs +4. Check artifacts and releases + +### Via GitHub CLI +```bash +# List recent workflow runs +gh run list --workflow=build-images.yml + +# Watch a running workflow +gh run watch + +# Download artifacts +gh run download +``` + +## Troubleshooting + +### Build Failures + +#### Stage 1 Failures +- **Symptom**: Download errors +- **Cause**: Network issues, broken URLs +- **Solution**: Check base image URLs in `autobuild` script + +#### Stage 2 Failures (QEMU) +- **Symptom**: QEMU timeout or setup.sh errors +- **Cause**: Package installation failures, network issues in QEMU +- **Solution**: Check `setup.sh` syntax, verify package names +- **Debugging**: Add `set -x` to `setup.sh` for verbose output + +#### Stage 3 Failures +- **Symptom**: Merge errors, loop device issues +- **Cause**: Insufficient permissions, partition layout issues +- **Solution**: Verify image formats, check merge script logic + +#### Stage 4 Failures +- **Symptom**: PiShrink errors +- **Cause**: Filesystem corruption, insufficient disk space +- **Solution**: Check stage 3 output, verify filesystem integrity + +### Disk Space Issues +- **Symptom**: "No space left on device" +- **Solution**: Already handled by "Free disk space" step +- **If persists**: Reduce IMAGE_SIZE in config.sh + +### Release Creation Failures +- **Symptom**: Permission denied errors +- **Cause**: Missing `contents: write` permission +- **Solution**: Verify workflow permissions in YAML + +## Customizing the Workflow + +### Adding a New Build Stage + +1. Add new job to `.github/workflows/build-images.yml` +2. Define dependencies with `needs: [previous-job]` +3. Add matrix strategy if parallel execution needed +4. Upload artifacts for next stage +5. Update artifact retention as needed + +### Changing Build Frequency + +Edit the `schedule` trigger: +```yaml +schedule: + # Build every 6 hours + - cron: '0 */6 * * *' +``` + +### Disabling Automatic Builds + +Comment out unwanted triggers: +```yaml +# on: +# push: +# branches: +# - '**' +# schedule: +# - cron: '0 2 * * *' + workflow_dispatch: # Keep manual trigger only +``` + +## Best Practices + +### For Image Developers +1. **Test locally first**: Run `./bin/autobuild --image ` before pushing +2. **Small iterations**: Commit small changes to debug issues faster +3. **Monitor logs**: Check Actions tab for build output +4. **Use branches**: Test in feature branches before merging to main + +### For Repository Maintainers +1. **Review PRs carefully**: Malicious setup.sh could compromise runners +2. **Monitor disk usage**: Adjust retention policies if needed +3. **Update base images**: Keep RaspiOS and Debian URLs current +4. **Clean old releases**: Remove outdated releases periodically + +## Security Considerations + +### Runner Security +- Workflows run in isolated GitHub-hosted runners +- Each job runs in a fresh VM +- Artifacts are sandboxed +- No access to repository secrets (unless explicitly added) + +### Image Security +- Setup scripts run with root privileges in QEMU +- Only trusted contributors should modify setup.sh +- Review all package installations +- Avoid hardcoded credentials + +### Release Security +- Releases are public by default +- Verify checksums before flashing images +- Use HTTPS for all downloads + +## Performance Metrics + +### Resource Usage (per image) +- **CPU**: 4 cores (QEMU) +- **RAM**: 8GB (QEMU) +- **Disk**: ~20GB peak usage +- **Network**: ~2GB download (base images) + +### GitHub Actions Limits +- **Concurrent jobs**: 20 (free tier) +- **Job timeout**: 6 hours (we set 60 min for QEMU) +- **Artifact size**: 10GB per file +- **Total storage**: 500MB artifacts (free tier) + +## Future Improvements + +Potential enhancements: + +1. **Caching base images**: Store RaspiOS/Debian images in GitHub cache +2. **Incremental builds**: Only rebuild changed images +3. **Build matrix**: Test multiple Debian versions +4. **Checksum verification**: Add SHA256 checksums to releases +5. **Build badges**: Display build status in README +6. **Multi-architecture**: Support x86_64 images for testing + +## Related Documentation + +- **[Home](Home)**: Project overview +- **[Main README](../README.md)**: Complete documentation +- **[CLAUDE.md](../CLAUDE.md)**: Technical architecture details +- **GitHub Actions**: [Official documentation](https://docs.github.com/en/actions) \ No newline at end of file diff --git a/Home.md b/Home.md index b332ed1..dcafa23 100644 --- a/Home.md +++ b/Home.md @@ -1 +1,223 @@ -Welcome to the raspberry-builds wiki! +# RPI-Dev - Hybrid Raspberry Pi Images Builder + +Welcome to the RPI-Dev wiki! This project automates the creation of hybrid Raspberry Pi images combining Raspberry Pi OS hardware support with custom Debian ARM64 root filesystems. + +## Overview + +RPI-Dev is an automated build system that creates ready-to-use Raspberry Pi images with full hardware compatibility and custom software configurations. The project addresses a common challenge: running pure Debian on Raspberry Pi while maintaining complete hardware support for critical components like the RP1 chip (Ethernet, USB, GPIO). + +### Key Features + +- **Automated Builds**: GitHub Actions automatically build all images daily and on every push +- **Full Hardware Support**: Maintains Raspberry Pi OS kernel and firmware for complete hardware compatibility +- **Pure Debian Userspace**: Uses official Debian ARM64 root filesystems for a clean, standard environment +- **Multi-Image Support**: Easy creation and management of multiple image configurations +- **Automatic Updates**: Built-in APT configuration enables safe kernel and firmware updates +- **QEMU Development**: Test and develop images in QEMU before flashing to hardware +- **CI/CD Integration**: Automated releases via GitHub Actions with compressed, ready-to-flash images + +## Why This Project? + +The Raspberry Pi uses specialized hardware that requires kernel drivers not yet available in mainline Linux: + +- **RP1 Southbridge Chip**: Handles Ethernet, USB 2.0/3.0, GPIO, and other critical I/O +- **VideoCore GPU**: Provides hardware acceleration and display output +- **WiFi/Bluetooth**: Requires specific firmware blobs + +Using a standard Debian kernel results in non-functional hardware. This project solves that by: + +1. Using Raspberry Pi OS boot partition and kernel +2. Replacing the root filesystem with pure Debian +3. Installing RaspiOS packages via APT for automatic updates +4. Preserving hardware compatibility while gaining Debian's advantages + +## How It Works + +### Architecture + +The project uses a **partition-level merge approach**: + +1. **Boot Partition (FAT32)**: Retained from Raspberry Pi OS for firmware compatibility +2. **Root Partition (ext4)**: Replaced with custom Debian ARM64 rootfs +3. **RaspiOS Packages**: Installed via APT with repository pinning: + - `raspberrypi-kernel` - Kernel, initramfs, and modules (including RP1 drivers) + - `raspberrypi-bootloader` - Bootloader and firmware files + - `libraspberrypi*` - VideoCore libraries + - `firmware-brcm80211` - WiFi/Bluetooth firmware +4. **APT Configuration**: RaspiOS repository with pinning enables safe `apt upgrade` + +### Build Process + +``` +┌─────────────────────────────────────────────────────────────┐ +│ Stage 1: Download Base Images │ +│ ├─ Raspberry Pi OS Lite ARM64 (.img.xz) │ +│ └─ Debian Generic Cloud ARM64 (.raw) │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Stage 2: QEMU Setup (Native ARM64 Execution) │ +│ ├─ Create setup.iso from image/setup.sh + setupfiles/ │ +│ ├─ Launch QEMU ARM64 with Debian + setup.iso │ +│ ├─ Execute setup.sh (install packages, configure system) │ +│ ├─ Install RaspiOS kernel/firmware via APT │ +│ └─ Automatic shutdown when complete │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Stage 3: Hybrid Image Creation │ +│ ├─ Copy RaspiOS boot partition │ +│ ├─ Replace root partition with configured Debian │ +│ ├─ Preserve /etc/fstab from RaspiOS │ +│ └─ Resize to final image size │ +└─────────────────────────────────────────────────────────────┘ + ↓ +┌─────────────────────────────────────────────────────────────┐ +│ Stage 4: Compression │ +│ ├─ Shrink unused space with PiShrink │ +│ └─ Compress to .img.xz (ready to flash) │ +└─────────────────────────────────────────────────────────────┘ +``` + +## Quick Start + +### For Users: Download Pre-Built Images + +The easiest way to use this project is to download pre-built images from [GitHub Releases](../../releases): + +1. Go to the [Releases](../../releases) page +2. Download the `.img.xz` file for your desired image +3. Flash to SD card or SSD: + ```bash + xz -dc rpi-*.img.xz | sudo dd of=/dev/sdX bs=4M status=progress conv=fsync + sync + ``` + +### For Developers: Fork and Customize + +Want to create your own custom images? It's easy: + +1. **Fork this repository** on GitHub +2. **Clone your fork**: + ```bash + git clone https://github.com/YOUR_USERNAME/rpi-dev.git + cd rpi-dev + ``` + +3. **Create a new image configuration**: + ```bash + # Copy an existing image as a template + cp -r images/raspivirt-incus images/my-custom-image + ``` + +4. **Edit the configuration** (`images/my-custom-image/config.sh`): + ```bash + OUTPUT_IMAGE="rpi-my-custom-image.img" + IMAGE_SIZE="8G" + QEMU_RAM="8G" + QEMU_CPUS="4" + DESCRIPTION="My custom Raspberry Pi image" + ``` + +5. **Customize the setup script** (`images/my-custom-image/setup.sh`): + - Add package installations + - Configure services + - Apply custom settings + +6. **Add custom files** to `images/my-custom-image/setupfiles/`: + - Configuration files + - Scripts + - SSH keys + +7. **Commit and push**: + ```bash + git add images/my-custom-image/ + git commit -m "Add my custom image" + git push origin main + ``` + +8. **GitHub Actions automatically builds your image!** + - Check the [Actions](../../actions) tab for build progress + - Download from [Releases](../../releases) when complete + +### Building Locally + +If you prefer to build locally: + +```bash +# Install dependencies +sudo apt install -y qemu-system-aarch64 qemu-utils parted \ + e2fsprogs dosfstools rsync xz-utils genisoimage + +# Build a specific image +./bin/autobuild --image my-custom-image + +# Or build all images +./bin/autobuild --all-images +``` + +## Image Directory Structure + +Each image is defined in `images//`: + +``` +images/my-image/ +├── config.sh # Build configuration +│ # - OUTPUT_IMAGE: Final image filename +│ # - IMAGE_SIZE: Final image size (e.g., "8G") +│ # - QEMU_RAM: RAM for QEMU (e.g., "8G") +│ # - QEMU_CPUS: CPU cores for QEMU (e.g., "4") +│ # - DESCRIPTION: Image description +│ +├── setup.sh # Setup script executed in QEMU ARM64 +│ # - Installs packages (RaspiOS kernel, software) +│ # - Configures system (users, services, etc.) +│ # - Runs in native ARM64 environment +│ +├── setupfiles/ # Files copied to /root/setupfiles/ in image +│ # - Config files, scripts, certificates, etc. +│ # - Available to setup.sh during QEMU execution +│ +└── cloudinit/ # Cloud-init configuration (for first boot) + ├── user-data # User configuration (users, SSH keys, passwords) + ├── meta-data # Instance metadata (hostname, instance-id) + └── seed.img # Auto-generated ISO (don't edit manually) +``` + +## Available Images + +This repository includes the following pre-configured images: + +- **[RaspiVirt-Incus](Image-RaspiVirt-Incus)**: Raspberry Pi virtualization platform with Incus container/VM manager +- **[RaspiVirt-Incus+Docker](Image-RaspiVirt-Incus-Docker)**: RaspiVirt-Incus plus Docker for container orchestration + +See individual image pages for detailed documentation. + +## Automatic Updates + +Images built with this system support automatic kernel and firmware updates via APT: + +```bash +# On your Raspberry Pi +sudo apt update +sudo apt upgrade -y +``` + +The APT pinning configuration ensures RaspiOS packages (kernel, firmware) are updated from the RaspiOS repository while all other packages use Debian repositories. This maintains hardware compatibility while keeping the system up-to-date. + +## Documentation + +- **[GitHub Actions Workflow](GitHub-Actions)**: Detailed documentation of the automated build system +- **[RaspiVirt-Incus Image](Image-RaspiVirt-Incus)**: Virtualization platform with Incus +- **[RaspiVirt-Incus+Docker Image](Image-RaspiVirt-Incus-Docker)**: Incus + Docker platform +- **[Main README](../README.md)**: Complete project documentation with manual build instructions + +## Getting Help + +- **Issues**: Report bugs or request features via [GitHub Issues](../../issues) +- **Discussions**: Ask questions in [GitHub Discussions](../../discussions) +- **Documentation**: Check the [README](../README.md) and [CLAUDE.md](../CLAUDE.md) for technical details + +## License + +This project is provided "as is" without warranty. Use at your own risk. \ No newline at end of file diff --git a/Image-RaspiVirt-Incus-Docker.md b/Image-RaspiVirt-Incus-Docker.md new file mode 100644 index 0000000..bd20dd5 --- /dev/null +++ b/Image-RaspiVirt-Incus-Docker.md @@ -0,0 +1,708 @@ +# RaspiVirt-Incus+Docker Image + +**RaspiVirt-Incus+Docker** extends the [RaspiVirt-Incus](Image-RaspiVirt-Incus) image by adding **Docker** container support alongside Incus. This image provides the best of both worlds: Incus for system containers and VMs, plus Docker for application containers with Docker Compose and the full Docker ecosystem. + +## Overview + +This image combines two powerful containerization platforms: +- **Incus** - System containers (LXC) and virtual machines (KVM) +- **Docker** - Application containers with OCI compatibility + +Additionally, the image includes: +- **Portainer** - Web-based Docker management UI +- **Watchtower** - Automatic Docker container updates + +### Key Features + +All features from [RaspiVirt-Incus](Image-RaspiVirt-Incus) and: +- **Docker Engine** - Latest Docker CE with containerd +- **Docker Compose** - Multi-container application orchestration (plugin v2) +- **Docker Buildx** - Advanced build features and multi-platform support +- **Portainer CE** - Web UI for Docker management (port 9443) +- **Watchtower** - Automatic container image updates (daily at 4 AM) +- **Dual Container Ecosystems** - Choose the right tool for each workload + +## Image Specifications + +- **Image Name**: `rpi-raspivirt-incus+docker.img.xz` +- **Base OS**: Debian 13 (Trixie) ARM64 +- **Kernel**: Raspberry Pi OS kernel (with RP1 drivers) +- **Image Size**: ~2.5 GB (expands on first boot) +- **Compressed Size**: ~700MB (xz compressed) + +### Build Configuration + +From `images/raspivirt-incus+docker/config.sh`: +```bash +OUTPUT_IMAGE="rpi-raspivirt-incus+docker.img" +IMAGE_SIZE="4G" +QEMU_RAM="8G" +QEMU_CPUS="4" +DESCRIPTION="Raspberry Pi image with Incus, KVM virtualization and br-wan bridge" +``` + +## Installed Software + +All packages from [RaspiVirt-Incus](Image-RaspiVirt-Incus) and: + +### Docker Stack +- **Docker CE** (`docker-ce`) - Docker Engine +- **Docker CLI** (`docker-ce-cli`) - Docker command-line interface +- **containerd** (`containerd.io`) - Container runtime +- **Docker Buildx** (`docker-buildx-plugin`) - Extended build capabilities +- **Docker Compose** (`docker-compose-plugin`) - Multi-container orchestration + +### Pre-Installed Containers + +#### Portainer CE (Latest LTS) +- **Purpose**: Web-based Docker management +- **Port**: 9443 (HTTPS), 8000 (Tunnel) +- **Volume**: `portainer_data` +- **Auto-start**: Yes +- **Image**: `portainer/portainer-ce:lts` + +#### Watchtower +- **Purpose**: Automatic container updates +- **Schedule**: Daily at 4:00 AM +- **Monitors**: All containers +- **Auto-start**: Yes +- **Image**: `containrrr/watchtower` + +## Docker Configuration + +### User Permissions + +The `pi` user is added to the `docker` group during setup: +```bash +usermod -aG docker pi +``` + +This allows running Docker commands without `sudo`: +```bash +# No sudo needed +docker ps +docker run hello-world +``` + +### Docker Daemon Configuration + +Override file created at `/etc/systemd/system/docker.service.d/override.conf`: +```ini +[Service] +Environment=DOCKER_MIN_API_VERSION=1.25 +``` + +This ensures compatibility with older Docker clients while maintaining security. + +### Docker Repository + +Official Docker repository configured at `/etc/apt/sources.list.d/docker.sources`: +``` +Types: deb +URIs: https://download.docker.com/linux/debian +Suites: trixie +Components: stable +Signed-By: /etc/apt/keyrings/docker.asc +``` + +Enables easy updates: +```bash +sudo apt update +sudo apt upgrade docker-ce docker-ce-cli containerd.io +``` + +## First-Boot Process + +Extends the [RaspiVirt-Incus first-boot process](Image-RaspiVirt-Incus#first-boot-process) with Docker initialization. + +### Stage 1: rpi-first-boot (Before Network) + +Identical to RaspiVirt-Incus: +1. Enable classic network names (`eth0`) +2. Disable cloud-init networking +3. Resize root partition +4. Deploy netplan configuration +5. Reboot + +### Stage 2: services-first-boot (After Network) + +Enhanced to include Docker initialization: + +**Script**: `/usr/local/bin/services-first-boot.sh` + +**Actions**: +1. **Wait for internet connectivity** (5 minute timeout) +2. **Initialize Incus**: + - Minimal init + web UI on :8443 + - Create `br-wan` network + - Attach to default profile +3. **Initialize Docker containers**: + - Create Portainer with persistent volume + - Create Watchtower with daily schedule +4. **Self-destruct** + +## Pre-Installed Containers + +### Portainer CE + +Portainer provides a comprehensive web UI for Docker management. + +#### Access Portainer + +1. Get Raspberry Pi IP: `ip addr show br-wan` +2. Open browser: `https://:9443` +3. Accept self-signed certificate +4. Create admin account on first login + +#### Portainer Features + +- **Container Management**: Start, stop, restart, delete containers +- **Image Management**: Pull, build, push images +- **Volume Management**: Create and manage volumes +- **Network Management**: Create and configure networks +- **Docker Compose**: Deploy stacks from compose files +- **Console Access**: Access container shells via web +- **Resource Monitoring**: CPU, memory, network usage +- **User Management**: Multi-user access with RBAC + +#### Portainer Configuration + +```bash +# Container details +docker inspect portainer + +# Ports: +# 8000 -> Tunnel server +# 9443 -> HTTPS web UI + +# Volumes: +# /var/run/docker.sock -> Docker API access +# portainer_data -> Persistent configuration + +# Restart policy: always +``` + +### Watchtower + +Watchtower automatically updates running Docker containers. + +#### How Watchtower Works + +1. Checks for new image versions daily at 4:00 AM +2. Pulls new images if available +3. Stops old containers gracefully +4. Starts new containers with same configuration +5. Cleans up old images + +#### Watchtower Configuration + +```bash +# Container details +docker inspect watchtower + +# Environment: +# WATCHTOWER_SCHEDULE: "0 0 4 * * *" (4:00 AM daily) + +# Monitored containers: +# All containers except those with label "hidden=true" +# (Portainer and Watchtower are hidden) + +# Restart policy: always +``` + +#### Controlling Watchtower + +```bash +# Update all containers immediately +docker restart watchtower + +# Exclude a container from updates +docker run -d --label com.centurylinklabs.watchtower.enable=false myimage + +# View Watchtower logs +docker logs watchtower +``` + +## Docker Usage Examples + +### Basic Commands + +```bash +# Check Docker version +docker --version + +# Check running containers +docker ps + +# Check all containers (including stopped) +docker ps -a + +# Pull an image +docker pull nginx:alpine + +# Run a simple container +docker run -d -p 80:80 nginx:alpine + +# View container logs +docker logs + +# Execute command in container +docker exec -it bash + +# Stop container +docker stop + +# Remove container +docker rm +``` + +### Docker Compose Example + +Create a `docker-compose.yml` file: +```yaml +services: + nginx: + image: nginx:alpine + ports: + - "80:80" + volumes: + - ./html:/usr/share/nginx/html + restart: unless-stopped + + redis: + image: redis:alpine + restart: unless-stopped +``` + +Deploy the stack: +```bash +# Start services +docker compose up -d + +# View logs +docker compose logs -f + +# Stop services +docker compose down +``` + +### Common Applications + +#### Web Server (Nginx) +```bash +docker run -d \ + --name nginx \ + -p 80:80 \ + -v /home/pi/www:/usr/share/nginx/html:ro \ + --restart unless-stopped \ + nginx:alpine +``` + +#### Database (PostgreSQL) +```bash +docker run -d \ + --name postgres \ + -p 5432:5432 \ + -e POSTGRES_PASSWORD=secretpassword \ + -v postgres_data:/var/lib/postgresql/data \ + --restart unless-stopped \ + postgres:alpine +``` + +#### Home Assistant +```bash +docker run -d \ + --name homeassistant \ + --privileged \ + --network host \ + -v /home/pi/homeassistant:/config \ + -e TZ=Europe/Paris \ + --restart unless-stopped \ + homeassistant/home-assistant:stable +``` + +#### Pi-hole (DNS + Ad Blocker) +```bash +docker run -d \ + --name pihole \ + -p 53:53/tcp -p 53:53/udp \ + -p 8080:80 \ + -e TZ=Europe/Paris \ + -e WEBPASSWORD=admin \ + -v pihole_etc:/etc/pihole \ + -v pihole_dnsmasq:/etc/dnsmasq.d \ + --restart unless-stopped \ + pihole/pihole:latest +``` + +## Incus + Docker Integration + +### When to Use Incus vs Docker + +#### Use Incus For: +- **System containers**: Full OS environments with systemd +- **Virtual machines**: When you need kernel isolation +- **Long-lived environments**: Development VMs, staging servers +- **Multi-distribution testing**: Run different Linux distros +- **Network isolation**: Complex network topologies + +#### Use Docker For: +- **Application containers**: Stateless microservices +- **Docker Compose stacks**: Multi-container applications +- **CI/CD**: Build and test pipelines +- **Pre-built images**: Leveraging Docker Hub ecosystem +- **Lightweight services**: Single-purpose containers + +### Shared Networking + +Both Incus and Docker containers can use the `br-wan` bridge: +- **Incus containers**: Automatically use `br-wan` via default profile +- **Docker containers**: Use host network mode for direct bridge access + +```bash +# Docker container on host network (uses br-wan) +docker run -d --network host nginx:alpine +``` + +## Network Configuration + +### Bridge Topology + +``` +Internet + ↓ +Your Router (DHCP) + ↓ +┌─────────────────────────────────────────┐ +│ Raspberry Pi │ +│ ┌───────────────────────────────────┐ │ +│ │ br-wan (Bridge) │ │ ← Gets IP from router +│ │ ├─ eth0 (Physical NIC) │ │ +│ │ ├─ Incus Container 1 │ │ ← Gets IP from router +│ │ ├─ Incus VM 1 │ │ ← Gets IP from router +│ │ └─ Docker (host network mode) │ │ ← Uses br-wan IP +│ └───────────────────────────────────┘ │ +│ │ +│ Docker (bridge network) │ +│ ┌───────────────────────────────────┐ │ +│ │ docker0 (172.17.0.0/16) │ │ +│ │ ├─ Portainer │ │ ← Internal Docker network +│ │ ├─ Watchtower │ │ ← Internal Docker network +│ │ └─ Your containers │ │ ← NAT to br-wan +│ └───────────────────────────────────┘ │ +└─────────────────────────────────────────┘ +``` + +### Accessing Services + +- **Incus Web UI**: `https://:8443` +- **Portainer**: `https://:9443` +- **Docker containers with published ports**: `http://:` +- **Incus containers**: Direct access via DHCP-assigned IPs + +## Use Cases + +All use cases from [RaspiVirt-Incus](Image-RaspiVirt-Incus#use-cases) plus: + +### Docker-Specific Use Cases + +#### Microservices Platform +- Deploy microservices with Docker Compose +- Use Incus for database VMs +- Portainer for central management + +#### Home Automation Hub +- Home Assistant in Docker +- Node-RED for automation +- MQTT broker for IoT +- InfluxDB + Grafana for monitoring + +#### Media Server +- Jellyfin/Plex in Docker +- Sonarr/Radarr for content management +- Transmission for downloads +- Storage in Incus container/VM + +#### Development Environment +- Application containers in Docker +- Database/services in Incus containers +- Isolated environments for each project + +## Customization + +Same customization options as [RaspiVirt-Incus](Image-RaspiVirt-Incus#customization) plus: + +### Modify Pre-Installed Containers + +Edit `setupfiles/services-first-boot.sh` to change Portainer/Watchtower configuration: + +```bash +# Example: Change Portainer to port 9000 +docker run -d \ + -p 8000:8000 -p 9000:9000 \ + --name portainer \ + --restart=always \ + -v /var/run/docker.sock:/var/run/docker.sock \ + -v portainer_data:/data \ + portainer/portainer-ce:lts +``` + +### Add Additional Pre-Installed Containers + +Add to `services-first-boot.sh` before the "Disable this service" section: + +```bash +# Create your custom container +echo " Creating custom container..." +docker run -d \ + --name myapp \ + -p 8080:8080 \ + --restart=always \ + myimage:latest +``` + +### Disable Portainer or Watchtower + +Comment out the respective sections in `services-first-boot.sh`: + +```bash +# # Create Portainer +# echo " Creating Portainer container..." +# docker volume create portainer_data +# ... +``` + +## System Resources + +### Resource Recommendations + +For optimal performance with both Incus and Docker: + +- **RAM**: 4GB minimum, 8GB recommended +- **Storage**: 32GB minimum, 64GB+ recommended +- **CPU**: Raspberry Pi 4 (4GB+) or Raspberry Pi 5 + +### Monitoring Resources + +```bash +# System resources +htop + +# Docker stats (real-time) +docker stats + +# Incus resource usage +incus info --resources + +# Disk usage +df -h +du -sh /var/lib/docker +du -sh /var/lib/incus +``` + +### Resource Limits + +Limit container resources to prevent overconsumption: + +```bash +# Docker: Limit container to 1GB RAM, 1 CPU +docker run -d \ + --memory=1g \ + --cpus=1 \ + nginx:alpine + +# Incus: Limit container to 2GB RAM, 2 CPUs +incus launch images:debian/13 limited \ + -c limits.memory=2GB \ + -c limits.cpu=2 +``` + +## Troubleshooting + +### Portainer Not Accessible + +**Check container status**: +```bash +docker ps | grep portainer +``` + +**View logs**: +```bash +docker logs portainer +``` + +**Restart Portainer**: +```bash +docker restart portainer +``` + +### Watchtower Not Updating Containers + +**Check schedule**: +```bash +docker inspect watchtower | grep WATCHTOWER_SCHEDULE +``` + +**View logs**: +```bash +docker logs watchtower +``` + +**Force update**: +```bash +# Trigger immediate update +docker restart watchtower +``` + +### Docker Daemon Not Starting + +**Check status**: +```bash +sudo systemctl status docker +``` + +**View logs**: +```bash +sudo journalctl -u docker -n 50 +``` + +**Restart Docker**: +```bash +sudo systemctl restart docker +``` + +### Permission Denied Errors + +**Verify user in docker group**: +```bash +groups pi +# Should include: pi sudo kvm incus incus-admin docker +``` + +**Re-login** if group membership was just added: +```bash +# Logout and login again, or: +newgrp docker +``` + +### Disk Space Issues + +**Check Docker disk usage**: +```bash +docker system df +``` + +**Clean up unused resources**: +```bash +# Remove unused images +docker image prune -a + +# Remove unused volumes +docker volume prune + +# Remove everything unused +docker system prune -a --volumes +``` + +## Security Considerations + +### Docker Security + +- **Change Portainer password** immediately after first login +- **Use secrets** for sensitive data (passwords, API keys) +- **Limit exposed ports** to only what's necessary +- **Use official images** from Docker Hub +- **Keep images updated** (Watchtower helps with this) +- **Avoid running privileged containers** unless required + +### Network Security + +- **Firewall**: Consider using ufw to restrict access +- **HTTPS**: Use reverse proxy (Traefik, Nginx) for HTTPS +- **VPN**: Access services via VPN instead of exposing to internet + +### Example: UFW Firewall + +```bash +# Install UFW +sudo apt install ufw + +# Allow SSH +sudo ufw allow 22/tcp + +# Allow Incus and Portainer locally only +sudo ufw allow from 192.168.1.0/24 to any port 8443 proto tcp +sudo ufw allow from 192.168.1.0/24 to any port 9443 proto tcp + +# Enable firewall +sudo ufw enable +``` + +## Performance Optimization + +### Docker Best Practices + +- Use **Alpine-based images** for smaller footprint +- Use **multi-stage builds** for efficient images +- Use **volume mounts** instead of copying large files +- Use **Docker Compose** for complex applications +- Use **health checks** for automatic restarts + +### Storage Optimization + +```bash +# Use overlay2 storage driver (default) +docker info | grep "Storage Driver" + +# Limit log size per container +docker run -d \ + --log-opt max-size=10m \ + --log-opt max-file=3 \ + nginx:alpine +``` + +## Package Updates + +Update both Debian/RaspiOS packages and Docker: + +```bash +# Update system packages +sudo apt update +sudo apt upgrade -y + +# Update Docker Engine +sudo apt install --only-upgrade \ + docker-ce \ + docker-ce-cli \ + containerd.io \ + docker-buildx-plugin \ + docker-compose-plugin + +# Update Incus +sudo apt install --only-upgrade incus incus-ui-canonical + +# Update containers (Watchtower does this automatically) +docker images | grep -v REPOSITORY | awk '{print $1}' | xargs -L1 docker pull +``` + +## Related Documentation + +- **[Home](Home)**: Project overview +- **[GitHub Actions](GitHub-Actions)**: Automated build system +- **[RaspiVirt-Incus](Image-RaspiVirt-Incus)**: Base image documentation +- **[Docker Documentation](https://docs.docker.com/)**: Official Docker docs +- **[Portainer Documentation](https://docs.portainer.io/)**: Portainer user guide +- **[Docker Compose Documentation](https://docs.docker.com/compose/)**: Compose reference + +## Build Information + +**GitHub Actions Workflow**: Automatically builds this image on push and daily schedule + +**Differences from RaspiVirt-Incus**: +- Adds Docker CE + plugins +- Adds Portainer and Watchtower containers +- Enhanced services-first-boot script + +**Download**: [Latest Release](../../releases) + +**Build Logs**: [GitHub Actions](../../actions) \ No newline at end of file diff --git a/Image-RaspiVirt-Incus.md b/Image-RaspiVirt-Incus.md new file mode 100644 index 0000000..cab2a3d --- /dev/null +++ b/Image-RaspiVirt-Incus.md @@ -0,0 +1,593 @@ +# RaspiVirt-Incus Image + +**RaspiVirt-Incus** is a Raspberry Pi image optimized for running containers and virtual machines using [Incus](https://linuxcontainers.org/incus/), the modern LXC/LXD fork. This image provides a complete virtualization platform with KVM support, bridged networking, and the Incus web UI. + +## Overview + +RaspiVirt-Incus transforms your Raspberry Pi into a powerful virtualization host capable of running: +- **System containers** (LXC) - Lightweight, fast containers with full OS isolation +- **Application containers** - OCI-compatible containers +- **Virtual machines** - KVM-accelerated VMs with full hardware virtualization + +### Key Features + +- **Incus Container Manager**: Modern LXC/LXD fork with active development +- **Incus Web UI**: Built-in web interface for easy management (port 8443) +- **KVM Virtualization**: Hardware-accelerated virtual machines on ARM64 +- **Bridged Networking**: `br-wan` bridge for direct network access to containers/VMs +- **Automatic Updates**: RaspiOS kernel + Debian packages via APT +- **First-Boot Configuration**: Automatic partition resize and network setup +- **Classic Network Names**: Uses `eth0` instead of `enp*` for predictable naming + +## Image Specifications + +- **Image Name**: `rpi-raspivirt-incus.img.xz` +- **Base OS**: Debian 13 (Trixie) ARM64 +- **Kernel**: Raspberry Pi OS kernel (with RP1 drivers) +- **Image Size**: ~1.5 GB (expands on first boot) +- **Compressed Size**: ~500MB (xz compressed) + +### Build Configuration + +From `images/raspivirt-incus/config.sh`: +```bash +OUTPUT_IMAGE="rpi-raspivirt-incus.img" +IMAGE_SIZE="4G" +QEMU_RAM="8G" +QEMU_CPUS="4" +DESCRIPTION="Raspberry Pi image with Incus, KVM virtualization and br-wan bridge" +``` + +## Installed Software + +### Core System +- **Debian 13 (Trixie)** - Latest Debian stable +- **Raspberry Pi Kernel** - Full hardware support (RP1, WiFi, Bluetooth) + - `linux-image-rpi-v8` - Raspberry Pi 3/4/5 kernel + - `linux-image-rpi-2712` - Raspberry Pi 5 optimized kernel + - `linux-headers-rpi-v8` - Kernel headers for module compilation + - `linux-headers-rpi-2712` - Pi 5 kernel headers + - `raspi-firmware` - Bootloader and firmware + - `firmware-brcm80211` - WiFi/Bluetooth firmware + +### Virtualization Stack +- **Incus** - Container and VM manager + - `incus` - Core daemon and CLI + - `incus-ui-canonical` - Official web UI +- **KVM/QEMU** - Hardware virtualization + - `qemu-system-aarch64` - ARM64 system emulator + - `qemu-kvm` - KVM acceleration support + - `qemu-utils` - Image utilities + - `qemu-efi-aarch64` - UEFI firmware for VMs + +### Networking +- **systemd-networkd** - Network management +- **netplan.io** - Network configuration +- **bridge-utils** - Bridge utilities +- **net-tools** - Classic networking tools (ifconfig, route) +- **iptables** - Firewall and NAT + +### System Utilities +- **curl**, **wget** - Download tools +- **sudo** - Privilege escalation +- **openssh-server** - Remote access +- **parted** - Partition management +- **ca-certificates**, **gnupg** - Security and package verification + +## Network Configuration + +### br-wan Bridge + +The image uses a **bridged network configuration** (`br-wan`) that allows containers and VMs to appear as physical devices on your network. + +#### Netplan Configuration (`/etc/netplan/99-br-wan.yaml`) + +```yaml +network: + version: 2 + renderer: networkd + ethernets: + eth0: + dhcp4: false + dhcp6: false + bridges: + br-wan: + interfaces: [eth0] + dhcp4: true + dhcp6: true +``` + +#### How It Works + +``` +Internet + ↓ +Your Router (DHCP) + ↓ +┌─────────────────────────────────┐ +│ Raspberry Pi │ +│ ┌───────────────────────────┐ │ +│ │ br-wan (Bridge) │ │ ← Gets IP from router +│ │ ├─ eth0 (Physical NIC) │ │ +│ │ ├─ Container 1 │ │ ← Gets IP from router +│ │ ├─ Container 2 │ │ ← Gets IP from router +│ │ └─ VM 1 │ │ ← Gets IP from router +│ └───────────────────────────┘ │ +└─────────────────────────────────┘ +``` + +**Advantages**: +- Containers/VMs get IPs directly from your router +- No NAT required +- Containers/VMs accessible from your LAN +- Simplified networking + +### Incus Network Integration + +Incus is configured to use the `br-wan` bridge in **passthrough mode**: +- Incus network name: `br-wan` +- Parent device: `br-wan` (system bridge) +- IPv4/IPv6: Disabled (managed by external DHCP) +- All containers/VMs attach to `br-wan` by default + +## First-Boot Process + +The image uses a **two-stage first-boot process** to configure the system: + +### Stage 1: rpi-first-boot (Before Network) + +Runs on first boot before network is configured. + +**Script**: `/usr/local/bin/rpi-first-boot.sh` + +**Actions**: +1. **Enable classic network names**: + - Adds `net.ifnames=0 biosdevname=0` to `/boot/firmware/cmdline.txt` + - Results in `eth0`, `wlan0` instead of `enp*`, `wlp*` +2. **Disable cloud-init networking**: + - Creates `/etc/cloud/cloud.cfg.d/99-disable-network-config.cfg` + - Prevents cloud-init from managing network (netplan takes over) +3. **Resize root partition**: + - Detects root partition + - Expands to use full SD card/SSD + - Resizes ext4 filesystem +4. **Deploy netplan configuration**: + - Moves `99-br-wan.yaml` to `/etc/netplan/` + - Generates netplan configuration +5. **Self-destruct**: + - Disables systemd service + - Deletes service file and script + - Reboots system + +### Stage 2: services-first-boot (After Network) + +Runs after reboot when network is available. + +**Script**: `/usr/local/bin/services-first-boot.sh` + +**Actions**: +1. **Wait for internet connectivity**: + - Pings 8.8.8.8 and 1.1.1.1 + - Timeout: 5 minutes + - Required for Incus initialization +2. **Initialize Incus**: + - Minimal initialization: `incus admin init --minimal` + - Configure HTTPS UI: `incus config set core.https_address :8443` + - Apply netplan (creates `br-wan` bridge) + - Create Incus network using `br-wan` + - Attach `br-wan` to default profile +3. **Self-destruct**: + - Disables systemd service + - Deletes service file and script + +**Why Two Stages?** +- Stage 1 runs **before network** to resize partition and configure network +- Stage 2 runs **after network** to initialize services requiring internet access +- Ensures proper ordering of operations + +## MOTD IP Updater + +The image includes a dynamic **Message of the Day (MOTD)** that displays network information on login. + +**Script**: `/usr/local/bin/update-motd-ip.sh` + +**Triggers**: +- On boot (`update-motd-ip.service`) +- On network changes (`update-motd-ip.path` monitors `/etc/netplan/`) + +**Displayed Information**: +- System hostname +- IP addresses (IPv4/IPv6) +- Network interfaces +- Incus web UI URL (https://IP:8443) + +## User Configuration + +### Default User + +- **Username**: `pi` +- **Password**: `raspberry` +- **Sudo**: Passwordless (`NOPASSWD:ALL`) +- **Groups**: `sudo`, `kvm`, `incus`, `incus-admin` + +**Cloud-Init Configuration** (`cloudinit/user-data`): +```yaml +users: + - name: pi + sudo: ['ALL=(ALL) NOPASSWD:ALL'] + shell: /bin/bash + lock_passwd: false + +chpasswd: + list: | + pi:raspberry + expire: false + +ssh_pwauth: true +``` + +### Security Recommendations + +After first boot, **immediately**: +1. Change the default password: `passwd pi` +2. Add SSH keys: `ssh-copy-id pi@` +3. Disable password authentication: Edit `/etc/ssh/sshd_config` +4. Configure Incus authentication (see below) + +## Incus Configuration + +### Accessing Incus Web UI + +1. Get Raspberry Pi IP address: `ip addr show br-wan` +2. Open browser: `https://:8443` +3. Accept self-signed certificate +4. Create admin account on first login + +### Basic Incus Commands + +```bash +# Check Incus status +incus info + +# List containers/VMs +incus list + +# Launch a container (gets IP from br-wan/DHCP) +incus launch images:debian/13 mycontainer + +# Launch a VM +incus launch images:debian/13 myvm --vm + +# Access container console +incus exec mycontainer -- bash + +# Stop container +incus stop mycontainer + +# Delete container +incus delete mycontainer +``` + +### Creating a Container with DHCP + +```bash +# Launch Debian container +incus launch images:debian/13 web1 + +# Check assigned IP (from your router) +incus list + +# Container is accessible from your LAN +ping +ssh user@ +``` + +### Creating a Virtual Machine + +```bash +# Launch Ubuntu VM with 2GB RAM, 2 CPUs +incus launch images:ubuntu/22.04 vm1 --vm \ + -c limits.memory=2GB \ + -c limits.cpu=2 + +# Access VM console +incus console vm1 + +# Access VM via SSH (after VM gets DHCP IP) +ssh user@ +``` + +### Network Configuration + +The default profile uses `br-wan`: +```bash +# View default profile +incus profile show default + +# Example output: +# devices: +# eth0: +# nictype: bridged +# parent: br-wan +# type: nic +``` + +All containers/VMs automatically get network access through `br-wan`. + +## System Optimization + +### Network Tuning (`/etc/sysctl.d/99-network-tuning.conf`) + +```bash +# Increased network buffer sizes for better performance +net.core.rmem_max = 134217728 +net.core.wmem_max = 134217728 +net.ipv4.tcp_rmem = 4096 87380 67108864 +net.ipv4.tcp_wmem = 4096 65536 67108864 + +# Enable IP forwarding for containers/VMs +net.ipv4.ip_forward = 1 +net.ipv6.conf.all.forwarding = 1 +``` + +### Timezone and Locale + +- **Timezone**: Europe/Paris (configurable in `setup.sh`) +- **Locale**: `fr_FR.UTF-8` (configurable in `setup.sh`) + +## Package Management + +### APT Repositories + +The image includes both **Debian** and **RaspiOS** repositories: + +**Debian** (`/etc/apt/sources.list`): +``` +deb http://deb.debian.org/debian trixie main contrib non-free +``` + +**RaspiOS** (`/etc/apt/sources.list.d/raspi.sources`): +``` +Types: deb +URIs: http://archive.raspberrypi.com/debian/ +Suites: trixie +Components: main +Signed-By: /usr/share/keyrings/raspberrypi-archive-keyring.pgp +``` + +**Incus** (`/etc/apt/sources.list.d/zabbly-incus-stable.sources`): +``` +Types: deb +URIs: https://pkgs.zabbly.com/incus/stable +Suites: trixie +Components: main +Signed-By: /etc/apt/keyrings/zabbly.asc +``` + +### APT Pinning (`/etc/apt/preferences.d/raspi-pin`) + +``` +# Pin RaspiOS packages for kernel/firmware/bootloader +Package: raspberrypi-kernel raspberrypi-bootloader libraspberrypi* firmware-brcm80211 +Pin: release o=Raspberry Pi Foundation +Pin-Priority: 1001 + +# Default Debian packages +Package: * +Pin: release o=Debian +Pin-Priority: 500 +``` + +This ensures: +- Kernel and firmware come from RaspiOS (hardware compatibility) +- All other packages come from Debian (stability and security) + +### Safe System Updates + +```bash +# Update all packages (Debian + RaspiOS kernel/firmware) +sudo apt update +sudo apt upgrade -y + +# Auto-upgrade without prompts +sudo apt full-upgrade -y +``` + +The pinning configuration prevents accidental kernel changes while allowing safe updates. + +## Use Cases + +### Home Lab Server +- Run multiple containers for different services +- Host web servers, databases, development environments +- Isolated environments for testing + +### Development Environment +- Create disposable development containers +- Test across multiple Linux distributions +- Develop and test ARM64 applications + +### Network Services +- DNS server (Pi-hole in container) +- VPN server (WireGuard/OpenVPN) +- Home automation (Home Assistant) + +### Education and Learning +- Learn containerization and virtualization +- Experiment with different Linux distributions +- Practice system administration + +## Customization + +### Modifying the Image + +1. **Fork this repository** +2. **Edit configuration** (`images/raspivirt-incus/config.sh`): + - Change image size + - Adjust QEMU resources +3. **Customize setup script** (`images/raspivirt-incus/setup.sh`): + - Add packages + - Change timezone/locale + - Install additional software +4. **Add custom files** (`images/raspivirt-incus/setupfiles/`): + - Configuration files + - Scripts + - Certificates +5. **Modify cloud-init** (`images/raspivirt-incus/cloudinit/user-data`): + - Change default user + - Add SSH keys + - Modify passwords +6. **Commit and push** - GitHub Actions builds automatically + +### Example Modifications + +#### Add Additional Packages + +Edit `setup.sh` around line 20: +```bash +apt install -y \ + curl \ + wget \ + sudo \ + openssh-server \ + vim \ + htop \ + docker.io # Add Docker +``` + +#### Change Default User + +Edit `cloudinit/user-data`: +```yaml +users: + - name: admin # Change username + sudo: ['ALL=(ALL) NOPASSWD:ALL'] + shell: /bin/bash + lock_passwd: false + +chpasswd: + list: | + admin:securepassword # Change password +``` + +#### Adjust Image Size + +Edit `config.sh`: +```bash +IMAGE_SIZE="16G" # Increase to 16GB +``` + +## Troubleshooting + +### Incus Web UI Not Accessible + +**Check Incus status**: +```bash +sudo systemctl status incus +``` + +**Verify HTTPS listener**: +```bash +incus config get core.https_address +# Should show: :8443 +``` + +**Check firewall** (if enabled): +```bash +sudo iptables -L -n | grep 8443 +``` + +### Containers Not Getting IP Addresses + +**Verify br-wan bridge**: +```bash +ip addr show br-wan +# Should show an IP address +``` + +**Check netplan**: +```bash +sudo netplan status +``` + +**Verify Incus network**: +```bash +incus network show br-wan +``` + +### First Boot Not Completing + +**Check logs**: +```bash +# Check rpi-first-boot service +journalctl -u rpi-first-boot.service + +# Check services-first-boot service +journalctl -u services-first-boot.service +``` + +**Common issues**: +- No internet connectivity (services-first-boot requires internet) +- DHCP not available on network +- Network cable not connected + +### Partition Not Resized + +**Manual resize**: +```bash +# Identify root partition +sudo fdisk -l + +# Resize partition (example: /dev/mmcblk0p2) +sudo parted /dev/mmcblk0 resizepart 2 100% + +# Resize filesystem +sudo resize2fs /dev/mmcblk0p2 +``` + +## Performance Tips + +### Use Fast Storage +- Use SSD instead of SD card (USB 3.0 or NVMe via PCIe) +- Enable TRIM for SSDs +- Use high-quality SD cards (A2 rating minimum) + +### Optimize for Containers +- Prefer containers over VMs (lower overhead) +- Use ZFS storage pool for better performance (optional) +- Limit container resources appropriately + +### Monitor System Resources +```bash +# Check CPU/RAM usage +htop + +# Check disk usage +df -h + +# Check Incus resource usage +incus info --resources +``` + +## Related Documentation + +- **[Home](Home)**: Project overview +- **[GitHub Actions](GitHub-Actions)**: Automated build system +- **[RaspiVirt-Incus+Docker](Image-RaspiVirt-Incus-Docker)**: This image plus Docker +- **[Incus Documentation](https://linuxcontainers.org/incus/docs/latest/)**: Official Incus docs +- **[Debian Documentation](https://www.debian.org/doc/)**: Debian reference + +## Build Information + +**GitHub Actions Workflow**: Automatically builds this image on push and daily schedule + +**Build Process**: +1. Download Raspberry Pi OS and Debian base images +2. Execute `setup.sh` in QEMU ARM64 +3. Install RaspiOS kernel and Incus via APT +4. Merge boot partition and rootfs +5. Compress with PiShrink + +**Download**: [Latest Release](../../releases) + +**Build Logs**: [GitHub Actions](../../actions) \ No newline at end of file diff --git a/_Sidebar.md b/_Sidebar.md new file mode 100644 index 0000000..aeea12b --- /dev/null +++ b/_Sidebar.md @@ -0,0 +1,25 @@ +## Navigation + +**Getting Started** +- [Home](Home) +- [Quick Start](Home#quick-start) + +**Documentation** +- [GitHub Actions](GitHub-Actions) +- [Build Process](GitHub-Actions#build-architecture) + +**Images** +- [RaspiVirt-Incus](Image-RaspiVirt-Incus) +- [RaspiVirt-Incus+Docker](Image-RaspiVirt-Incus-Docker) + +**Resources** +- [Main README](https://github.com/Pikatsuto/raspberry-builds/blob/main/README.md) +- [Releases](https://github.com/Pikatsuto/raspberry-builds/releases) +- [Issues](https://github.com/Pikatsuto/raspberry-builds/issues) + +--- + +**Quick Links** +- [Download Images](https://github.com/Pikatsuto/raspberry-builds/releases) +- [Report Bug](https://github.com/Pikatsuto/raspberry-builds/issues/new) +- [GitHub Actions](https://github.com/Pikatsuto/raspberry-builds/actions) \ No newline at end of file