Most homelabs end up with a Discord server, a Slack workspace, or a family group chat scattered across three different apps. If you already run Proxmox VE for your VMs and containers, there's a decent chance you'd rather keep your chat data on your own hardware too. Rocket.Chat is an open-source, self-hosted alternative to Slack or Microsoft Teams, and it runs comfortably inside a single LXC container on Proxmox VE.
This guide walks you through the whole process: creating the container, installing Docker inside it, and deploying Rocket.Chat with its MongoDB database. You don't need any prior Docker experience, and every command is explained as you go.
What You Will Learn
- What Rocket.Chat is and why homelab users and small teams run it themselves
- How to create a Debian-based LXC container sized correctly for Rocket.Chat
- How to install Docker Engine inside an unprivileged LXC container
- How to deploy Rocket.Chat and MongoDB using Docker Compose
- How to complete the first-run admin setup and connect from your network
- How to fix the errors people run into most often during this install
What Is This Feature?
Rocket.Chat is a self-hosted team messaging platform. Think channels, direct messages, file sharing, and video calls, similar to Slack, except the entire application and its database live on a server you control. There's no per-user monthly fee and no third party reading your message history.
To run it, we're going to lean on two things Proxmox VE already gives you. The first is LXC (Linux Containers), a lightweight way to run an isolated Linux environment that shares the host's kernel instead of virtualizing a whole machine. Containers start in seconds and use a fraction of the RAM a full VM would need for the same job. The second is Docker, a tool that packages an application together with everything it needs to run into a single, repeatable unit called a container image. Rocket.Chat's developers publish official Docker images, which means you get the exact software they tested, not a version you pieced together from a wiki page.
Yes, this means you'll have Docker containers running inside an LXC container. That sounds odd the first time you hear it, but it's a well-worn pattern in the Proxmox community and works reliably as long as the LXC container is unprivileged with nesting enabled, which we'll set up in Step 1.
Why Would You Use It?
If you're already self-hosting things like Nextcloud or Vaultwarden, Rocket.Chat is a natural next step for keeping team or family communication in-house as well. A few common reasons people set it up:
- Coordinating a small business or side project without paying per-seat SaaS pricing
- Running a private community or Discord alternative you fully control
- Integrating chat notifications from home automation, monitoring tools, or CI pipelines through its webhook and bot support
- Keeping sensitive conversations off third-party servers entirely
Honestly, if you just want a quick group chat for two people, this is overkill — a group chat app on your phone will do. Rocket.Chat earns its keep once you have a handful of users, channels, and integrations to manage.
Prerequisites
Before you start, make sure you have:
- Proxmox VE 8.x or 9.x installed and reachable from your browser
- At least 4 GB of free RAM and 20 GB of free storage on the node (Rocket.Chat plus MongoDB is heavier than most self-hosted apps)
- A Debian 13 (or Debian 12) LXC template downloaded, or access to download one from the Proxmox template list
- Basic comfort typing commands in a Linux shell — you'll copy and paste most of them anyway
- Root or an account with permission to create containers on the node
Step-by-Step Tutorial
Step 1: Create the LXC container
Log in to the Proxmox VE shell (or use the web GUI's Create CT wizard) and create an unprivileged container. Nesting has to be enabled, since it lets Docker run its own containers and namespaces inside our LXC container.
pct create 220 local:vztmpl/debian-13-standard_13.1-1_amd64.tar.zst \
--hostname rocketchat \
--cores 4 \
--memory 4096 \
--swap 512 \
--rootfs local-lvm:16 \
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
--unprivileged 1 \
--features nesting=1,keyctl=1
pct start 220Adjust the template filename to match whatever version you actually downloaded — check with pveam list local if you're not sure what's available. The --features nesting=1,keyctl=1 flag is the part people forget, and skipping it is the single most common reason Docker refuses to start in the next step.
Step 2: Log in and update the container
Open a console to the new container from the Proxmox GUI, or SSH in once it has an IP address.
pct enter 220
apt update && apt full-upgrade -y
apt install -y ca-certificates curl gnupgRunning the update before installing anything else avoids dependency conflicts later, and it's a habit worth keeping for every container you spin up.
Step 3: Install Docker Engine
We'll add Docker's official APT repository rather than using the version bundled with Debian, since it tends to be several releases behind.
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/debian/gpg -o /etc/apt/keyrings/docker.asc
chmod a+r /etc/apt/keyrings/docker.asc
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/debian \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
tee /etc/apt/sources.list.d/docker.list > /dev/null
apt update
apt install -y docker-ce docker-ce-cli containerd.io docker-compose-pluginOnce that finishes, confirm Docker is actually working before moving on:
docker run hello-worldIf that prints a friendly welcome message, nesting is working correctly and you're clear to continue. If it hangs or errors out, jump ahead to the Troubleshooting section — don't waste time deploying Rocket.Chat on top of a broken Docker install.
Step 4: Write the Docker Compose file
Create a folder to hold Rocket.Chat's configuration and data, then write the compose file.
mkdir -p /opt/rocketchat && cd /opt/rocketchat
nano docker-compose.ymlPaste in the following. This is the same layout Rocket.Chat documents officially: one container for the app, one for MongoDB, and a small helper container that initializes MongoDB's replica set (Rocket.Chat needs this to watch the database for changes in real time).
services:
rocketchat:
image: rocket.chat:latest
restart: unless-stopped
volumes:
- ./uploads:/app/uploads
environment:
- PORT=3000
- ROOT_URL=http://your-server-ip:3000
- MONGO_URL=mongodb://mongodb:27017/rocketchat?replicaSet=rs0
- MONGO_OPLOG_URL=mongodb://mongodb:27017/local?replicaSet=rs0
depends_on:
- mongodb
ports:
- 3000:3000
mongodb:
image: mongo:6.0
restart: unless-stopped
volumes:
- ./data/db:/data/db
command: mongod --oplogSize 128 --replSet rs0
mongo-init-replica:
image: mongo:6.0
command: >
mongosh mongodb://mongodb:27017 --eval
'
rs.initiate({
_id: "rs0",
members: [ { _id: 0, host: "mongodb:27017" } ]
})
'
depends_on:
- mongodbReplace your-server-ip in the ROOT_URL line with the container's actual IP address or a domain name if you're putting a reverse proxy in front of it later. Rocket.Chat uses this value to build links inside emails and notifications, so it's worth getting right from the start.
Step 5: Start the stack
docker compose up -dGive it a minute or two on first boot — MongoDB has to initialize its replica set and Rocket.Chat has to run its own database migrations before the web interface responds. You can watch progress with:
docker compose logs -f rocketchatOnce you see a line mentioning the app is listening on port 3000, it's ready.
Step 6: Finish setup in the browser
Open http://<container-ip>:3000 in a browser on the same network. Rocket.Chat's setup wizard will ask you to create the first admin account, name your organization, and pick which optional features to enable. None of this is permanent — you can change all of it later from the admin panel.
Commands Explained
| Command | What it does |
|---|---|
pct create ... --features nesting=1,keyctl=1 | Creates the container and allows it to run its own container runtime (Docker) inside it |
docker run hello-world | A tiny test image that confirms Docker can actually pull and run containers before you trust it with something bigger |
docker compose up -d | Reads docker-compose.yml and starts every service it defines in the background (-d for detached) |
docker compose logs -f rocketchat | Streams live log output from just the Rocket.Chat container, useful for watching startup progress or diagnosing errors |
mongod --oplogSize 128 --replSet rs0 | Starts MongoDB with a replica set named rs0, which Rocket.Chat requires to receive real-time database updates |
Common Errors
| Error | Cause |
|---|---|
| docker: Error response from daemon: could not select device driver | Nesting wasn't enabled on the LXC container, or the container is privileged when it shouldn't need to be |
| MongoServerSelectionError: connect ECONNREFUSED | Rocket.Chat started before MongoDB finished initializing its replica set — usually resolves itself within a minute, or means the mongo-init-replica step never ran |
| Page loads but login redirects in a loop | The ROOT_URL value doesn't match the address you're actually browsing to |
| Container has no internet access during apt update | The bridge or DHCP settings on --net0 aren't matching your actual network — see the general Proxmox VE networking troubleshooting guide |
Troubleshooting
If docker run hello-world fails inside the container, stop and go back to the container's options in the Proxmox GUI. Check the Features tab and confirm Nesting is ticked. If you created the container with the CLI, double check the exact flag with:
pct config 220 | grep featuresYou should see nesting=1 in the output. If it's missing, you can add it without recreating the container:
pct set 220 --features nesting=1,keyctl=1
pct reboot 220If the web interface never loads at all, check that the container actually got an IP address and that port 3000 isn't already used by something else on that container:
ip a
ss -tulpn | grep 3000If MongoDB keeps restarting, it's almost always a permissions problem on the ./data/db volume folder. Unprivileged containers map root inside the container to an unprivileged UID on the host, and MongoDB's official image is picky about file ownership. Deleting the folder and letting Docker recreate it usually clears this up, assuming you haven't put real data in there yet.
Best Practices
- Put a reverse proxy (Nginx Proxy Manager or Caddy) in front of Rocket.Chat and get it on HTTPS before inviting anyone else to use it — chat credentials shouldn't travel in plain text
- Back up the
/opt/rocketchat/data/dband/opt/rocketchat/uploadsfolders, not just the container itself — that's where your actual messages and files live - Set resource limits on the LXC container (CPU and memory) once you know your real usage, rather than leaving it unbounded on a shared node
- Pin the Rocket.Chat image to a specific version tag instead of
latestonce you're running it for real — this stops an unexpected upgrade from breaking your setup overnight - Snapshot the container before every Rocket.Chat or MongoDB version upgrade
Frequently Asked Questions
Can I run Rocket.Chat without Docker, directly on the LXC container?
Yes, but you'd be building from source with Meteor and managing MongoDB manually, which is significantly more work to maintain. Docker is the officially supported path and the one this guide follows.
How much RAM does Rocket.Chat actually need for a handful of users?
4 GB is comfortable for a small team of five to twenty people. MongoDB is the bigger consumer of the two services, not Rocket.Chat itself.
Can I use an existing MongoDB server instead of running one in this container?
Yes — just point MONGO_URL and MONGO_OPLOG_URL at your external database and remove the mongodb and mongo-init-replica services from the compose file.
Does Rocket.Chat support video calls out of the box?
Basic one-to-one video calling works out of the box using WebRTC. Group video calls require connecting an external Jitsi server, which is a separate setup.
Is this safe to expose directly to the internet?
Not without a reverse proxy handling TLS and, ideally, a firewall rule limiting which IPs can reach port 3000 directly. Treat it the same way you'd treat any other self-hosted login page.
What happens if I reboot the LXC container?
Docker's restart: unless-stopped policy in the compose file brings both containers back up automatically once the LXC container finishes booting. You don't need to run docker compose up again manually.
Conclusion
You now have a working Rocket.Chat instance running entirely on hardware you control, tucked inside a single lightweight LXC container. From here, the natural next steps are putting it behind a reverse proxy with a real domain and HTTPS, inviting your team or household, and exploring the admin panel's integrations if you want chat notifications from your other homelab services. The container itself uses barely any resources at idle, so it's a reasonable thing to leave running alongside everything else on your node.