Disaster averted

Data loss didn't happen!

So I've ranted, raved, and preached ZFS for so many years, and it continues to pay off! After many years of running my main array, I finally lost a drive.

For many years now, I've been running a set of 3, 10TB drives in a RAIDZ, and I later added another 3 12TB drives, also in a RAIDZ, I was smart enough to buy 4 drives when I added those, and left the last in a a hot spare. Low and behold, I woke up to a message on the 27th stating that I had no more hot spare, and my array was resilvering. My 10TB had enough errors, and was booted from the array, and replaced with the hot spare, all while I was sleeping.

All data is safe

Not only did the hot spare mean that there was zero downtime while this all happened, there were also 2 more drives that could have died before data was lost, and even then, proper backups have been made. There is a full backup of all non replacable data as well as full machine backups on a schedule on a seperate machine locally, as well as offsite storage of critical data. RAID is NOT A BACKUP. That said, I love my uptime, and don't want the massive inconvenience of fixing things. Praise ZFS, and all it's holy glory.

Planned downtime.

While I could absolutely pull the old drive, and replace it with this new hot space that I've obtained, I'm going to take this chance to pull the machine down, dust it, and do host level upgrades. Downtime should be no more than 2 hours, and will be over this upcoming weekend as of the time of this post. Keep an eye on https://status.kdb424.xyz if you need updates on what's going on, or just want to know when things are happening. I keep a good log of when anything is happening there, as well as scheduled maintenance times.

Running all of this is getting expensive!

Self hosting is absolutely cheaper than paying for services elsewhere, and more importantly, the data stays on trusted servers. I'm not the only one using these services anymore! Friends are using immich, my git server, XMPP (which is starting to stockpile a bit of storage usage for media that people send), as well as other services. We're currently using 73.4% of the ~40TB capacity, and that's logical capacity. There's nearly 80TB of physical storage in the server, not to mention the 32TB of backup disks, not to mention the offsite storage. I'm also paying ~$13USD/month to make things available to others via VPS, and that's ignoring the electricity to power all of this storage, and an Intel GPU for transcoding video on the fly (media streaming, and compression for immich videos). If you're using these services, and would like to see me do more, consider buying me a coffee. I never like to ask, but I've been offering more and more to people, and costs are adding up! That said, don't be scared to reach out and see if you can have an account on anything. Worst you'll get is a polite no :)

Movim, the XMPP client and blogging platform!

Movim is up!

I wanted to get completely off of other's hosted platforms as much as I could, and noticed that there weren't always great clients for XMPP on some platforms, as they were all missing this or that feature, so I decided to host a movim!

Micro blogging, or just chatting with friends in the usual way, it's got something for everyone. Feel free to check out my instance at https://movim.kittie.gay or you can host your own! Speaking of which...

How to host it!

As usual, I'm going to assume you know about reverse proxies, DNS, and docker, and just leave the goods.

# docker-compose.yml
services:
  movim:
    image: ravermeister/movim:latest
    container_name: movim
    volumes:
      - .env:/usr/local/share/movim/.env:ro
    ports:
      - 8075:80
    restart: always
    depends_on:
      postgresql:
        condition: service_started

  postgresql:
    image: postgres:17.9-alpine
    hostname: postgresql
    container_name: postgresql
    env_file:
      - .env
    volumes:
      - database:/var/lib/postgresql/data:rw
    ports:
      - 9758:5432
    restart: always

volumes:
  database:
# .env

# Database configuration
DB_DRIVER=pgsql
DB_HOST=fry.home
DB_PORT=9758
DB_DATABASE=movim
DB_USERNAME=movim
DB_PASSWORD=SuperSecretPassword

# Daemon configuration
DAEMON_URL=https://movim.kittie.gay/ # Public URL of your Movim instance
DAEMON_PORT=8080 # Port on which the daemon will listen
DAEMON_INTERFACE=127.0.0.1 # Interface on which the daemon will listen, must be an IP
DAEMON_DEBUG=false
DAEMON_VERBOSE=false

# POSTGRESQL container variables
POSTGRES_DB=movim
POSTGRES_USER=movim
POSTGRES_PASSWORD=SuperSecretPassword

The end of another chat era

It's time to leave Discord!

With Discord ensuring it's own destruction, between collecting user ID's, to massive machine learning on it's users, to probable ties to palantir beyond what they claim, I'm done with Discord.

Over the years, I've moved from IRC, to many platforms that used XMPP (Google talk, MSN Messenger, AIM, Yahoo, Facebook, probably more), to Skype, to Discord, and I even tried to like Matrix on the way. While Matrix never fit the bill for me for several reasons, XMPP seems to have what I'm looking for, so that's where I'll be with the death of Discord.

So how do you reach me now?

I'm glad you asked. I'm currently running my own XMPP server! I welcome anyone here to join it by shooting me a message anywhere you currently have contact with me. If you can't get in contact the usual ways, please send an email to help AT kdb424.xyz. Please format that as an email, I'm trying to prevent spam, which is why it's formatted that way.

If you want to get a client downloaded to connect to the chat server while waiting for me to make you an account, I've blogged about those here.

If you're already on xmpp, or already have an account.

I'm on IRC as katelin AT kittie.gay and the same as with emails, please use the @ symbol, and no spaces. It's formatted the same as an email. The XMPP server federates with other servers, so I should be able to chat with you anywhere! Just don't forget to mention to me who you are, as I have historically been spammed on chat services in general, and am used to ignoring people.

XMPP - How to connect!

So you want to talk on XMPP, huh?

Welp, how do you do it?! Unlike most chat services you may be used to, XMPP is a protocol. There are many ways to connect to it, which is a blessing and a curse. It's nice to have options, but sometimes it's too much to look through if you don't know what there is out there. Here are my recommended clients for each platform to help get you started.

Gajim (Windows/Mac/Linux)

This is my starting recommendation for Windows and Linux. It supports most modern features, and has a decent interface. It's what I personally use on my computer at the moment! It is missing calling though, which is a big no-go for many though.

Kaidan (Linux)

I don't personally enjoy the UX of this one as much, but it does seem to support most things, calling included! If this is important to you, check this out.

Conversations (Android)

This is a great client that supports everything most people will want! If it's free in the play store, I'd grab in from there, or if they decide to charge for it there, it's around for free in the F-Droid store as it's open source!

Cheogram (Android)

Very similar to Conversations, but with some extra features. It's also in the play store, as well as F-Droid, and open source. This should also not cost you any money.

Monal IM (Mac OS/iOS)

If you're using an Apple device, this is the one for you. A beautiful UI, does everything you could ask of XMPP. Non Apple users are jealous of this option, what else can be said?

Movim (Web browser, works everywhere)

We have Movim support now on kittie.gay! Feel free to log into this platform for not only chatting, but micro blogging! It's very similar to a teeny twitter/bluesky/mastodon, but all over XMPP on the same account! Not sure how much I'll be micro blogging, but it's there if you want to check it out!

Movim official site

What else?

If the above is not your speed, feel free to search "XMPP client [platform here]" or check here for a decent sorted list from the XMPP official website!

XMPP - Taking back control of chats

With Discord in talks with Palantir, it's time to move on from having my chats controlled by a platform that's only going to continue to go down hill. We've been through many a service through the years, from things like AIM, MSN Messanger, Gtalk, and mumble, to Skype, eventually Discord, and probably other platforms that I've forgotten about. I'm well versed in self hosting at this point, and am done with changing platforms. It's time for self hosting a chat service.

Why XMPP/Jabber?

While XMPP is a very old spec in terms of when it came out, it's been wildly used since by many people, and kept alive with new specs. It's also common to run an ICE/STUN/TURN server next to it for voice and video. It's allows for federation, but doesn't require it (Lookin at you Matrix), it's relatively easy to self host, with limited resources needed, and only a bit of docker knowledge to make it easy.

So how do we do it?

While I'm running my own server that I'll mention more about at the end, I encourage anyone to self host if you can. That takes control back of your data, and lets you have control of reliability, ect. While my way isn't the only way, this is how I did it...

DNS

This has been the absolute worst part for me, and I'm sure there's some sort of issue here, but it's working well for me currently. You'll want a domain name, not a subdomain for hosting the core XMPP service, or you'll have additional steps and headaches with DNS, but if you know better than I, you don't need me to tell you this. Here's what I have for DNS.

DNS

I'm not going to go over this as I won't at all claim to be an expert, but this is what I've found works great, and seems to follow the documentation for several IRC servers.

Docker

services:
  prosody:
    image: prosodyim/prosody:13.0
    container_name: xmpp
    restart: unless-stopped
    pull_policy: always
    ports:
      - "5222:5222" # C2S STARTLS
      - "5269:5269" # S2S STARTTLS
      - "5280:5280" # HTTP (goes to CADDY)
      - "5223:5223" # legacy
      - "5000:5000" # file transfer proxy
    volumes:
      - data:/var/lib/prosody
      - modules:/ets/prosody/modules
      - ./prosody.cfg.lua:/etc/prosody/prosody.cfg.lua:ro
      - certs:/etc/prosody/certs:ro

  coturn:
    image: coturn/coturn:4.9.0
    container_name: coturn
    restart: unless-stopped
    network_mode: host
    volumes:
      - ./turnserver.conf:/etc/coturn/turnserver.conf:ro
    tmpfs:
      - /var/lib/coturn
    
volumes:
    certs: # TLS certs
    data: # most server data
    modules: # prosody modules. Can be omitted if you want to use a stock setup

If you know what a compose file looks like, this all should be self-explanitory. We run both a prosody server (xmpp/jabber), and a coturn server in tandum to offer voice/video/screenshare to xmpp clients, and the core spec doesn't allow for that. Coturn give us that for free!

Config files passed through

--- procody.cfg.lua
modules_enabled = {
    -- Core
    "roster";  -- Manage and store client rosters
    "saslauth"; -- Authentication using SASL
    "tls"; -- Support for SSL/TLS encryption
    "dialback"; -- Dialback support for server-to-server identity verification
    "disco"; -- Service Discovery support
    "ping"; -- XMPP Ping reply support
    "register"; -- No clue
    "time"; -- Reply to “What time is it?” requests
    "uptime"; -- Reply to uptime requests
    "version"; -- Reply to software version requests

    -- Security
    "blocklist";

    -- Multi-device & mobile
    "carbons"; "csi_simple";
    "smacks";         -- Stream Management (reliable delivery)
    "cloud_notify";   -- Push notifications for mobile

    -- Message archive
    "mam";

    -- User profiles & presence
    "vcard_legacy"; "pep"; "bookmarks";

    -- Admin
    "admin_shell";
    "admin_adhoc"; -- Allows administration via an XMPP client that supports ad-hoc commands
    -- Coturn
    "external_services";
    "turn_external";

    -- Contact info
    "server_info";
    "server_contact_info";

    -- Compliance
    "bosh"; -- Enable mod_bosh
    "websocket"; -- Enable mod_websocket
    "pubsub_serverinfo";
    "seclabels";

}

-- -----------------------------------------------------------------
-- Connection-level rate limiting
-- -----------------------------------------------------------------

limits = {
  c2s = {
    rate = "10kb/s";
  };
  s2sin = {
    rate = "30kb/s";
  };
}
--
-- -----------------------------------------------------------------
-- muc_mam - Message archiving for Multi-User Chat - XEP-0313
-- -----------------------------------------------------------------

muc_log_by_default = false
muc_log_presences = false
log_all_rooms = false
muc_log_expires_after = "2w"

c2s_require_encryption = true
s2s_require_encryption = true
s2s_secure_auth = true
authentication = "internal_hashed"
allow_registration = false
archive_expires_after = "1y"
default_archive_policy = true
http_interfaces = { "*" }
http_ports = { 5280 }
https_ports = { } -- reverse proxied through caddy on port 80

------------------------------------------------------------------------

VirtualHost "kittie.gay"

turn_external_host = "kittie.gay"
turn_external_port = 3478
turn_external_secret = "TURN SECRET GOES HERE"

admins = { "katelin@kittie.gay" }

disco_items = {
	{"kittie.gay", "primary servers"};
	{"proxy.kittie.gay", "proxy service for clients that need it"};
}

-- -----------------------------------------------------------------
-- Contact info - XEP-0157
-- -----------------------------------------------------------------

server_info = {
  abuse = { "mailto:abuse@kittie.gay", "xmpp:admin@kittie.gay" };
  admin = { "mailto:admin@kittie.gay", "xmpp:admin@kittie.gay" };
  security = { "xmpp:katelin@kittie.gay" };
  support = { "xmpp:support@muc.kittie.gay?join" };
}


-- -----------------------------------------------------------------
--- Set up a MUC (multi-user chat) room server
-- -----------------------------------------------------------------
Component "conference.kittie.gay" "muc"
    modules_enabled = { "muc_mam"; "mam"; }
    restrict_room_creation = "local"

Component "upload.kittie.gay" "http_file_share"
    http_file_share_size_limit = 10485760    -- 10 MB
    http_file_share_expires_after = 2592000  -- 30 days
    http_host = "kittie.gay"
    http_external_url = "https://kittie.gay"
    modules_disabled = {
        "s2s";
    }

Component "proxy.kittie.gay" "proxy65"
    proxy65_address = "xmpp.kittie.gay"
    modules_disabled = {
        "s2s";
    }

------------------------------------------------------------------------

Include "conf.d/*.cfg.lua"
#turnserver.conf
listening-port=3478
tls-listening-port=5349
min-port=49152
max-port=49200
relay-threads=2
realm=kittie.gay
use-auth-secret
static-auth-secret=TURN_SECRET_GOES_HERE
no-multicast-peers
no-cli
no-tlsv1
no-tlsv1_1
denied-peer-ip=10.0.0.0-10.255.255.255
denied-peer-ip=172.16.0.0-172.31.255.255
denied-peer-ip=192.168.0.0-192.168.255.255
log-file=stdout

With these files, the only thing missing is certs. I'll leave that one up to you, as everyone does theirs differently, and if you're running something on port 80/443, it can be a bit different. Check out some of the additional reading links for how others have done this.

This server with the configs above, covers 95% of compliance.

compliance

What's next?

Well, host your own, or let me know if you want an account! I'd gladly make anyone on, as I have registrations disabled for abuse reasons, but I want people to come over and say hi, federated or otherwise. I'm always around at help [at] kittie.gay, obviously with a symbol for @, but bot spam is crazy these days. You can also find me on XMPP at katelin [at] kittie.gay as well as the other usual suspects.

I'll have a follow up blog post recommending different clients for different platforms. I want to do a little more testing before I make that post though!

Happy chatting <3

Additional reading.

https://gsvd.dev/blog/my-prosody-setup-with-docker-compose

https://blog.dmcc.io/journal/xmpp-turn-stun-coturn-prosody/

https://compliance.conversations.im

https://github.com/jjj333-p/configs/blob/main/prosody.cfg.lua

https://github.com/monal-im/Monal/wiki/Considerations-for-XMPP-server-admins

PS

If you're wondering what the compliance of my server is at any time, feel free to see a live read below.