Machine translation from Turkish (Cloudflare Workers AI), not reviewed by the author. Read the Turkish original

← Posts

End-to-End Network Optimization with Antigravity and the Council, OPNsense Configuration, and the New Kastamonu İnci Dormitory Setup

Score —

The Linux kernel and BBR network optimization we carried out with Google Antigravity and a 12-member artificial intelligence council, the FQ_CoDel and zero-latency DNS architecture on OPNsense, and Kastamonu İnci Dormitory's new setup based on UniFi Wi-Fi 6, 5651, and Cloudflare edge computing.

· Updated: · 15 min read
Contents

Summary: A high-bandwidth fiber internet subscription alone does not mean low latency, a stable connection, or a secure network. From core queue algorithms to PPPoE packet size (MTU/MSS) calculations, from DNS leaks to VLAN isolation in multi-story buildings, if every layer is not configured correctly, even a 1000 Mbps line experiences momentary drops, bufferbloat, and serious security vulnerabilities. In this article, I first cover the fundamental engineering principles and security measures that must be known when building or improving a network. Then I share the end-to-end optimization we carried out with Google Antigravity and our 12-member artificial intelligence council on our own Linux workstation and on our OPNsense (FreeBSD 15.1) firewall, and finally the new structure we built for Kastamonu Özel İnci Yükseköğrenim Kız Öğrenci Yurdu (kastamonuinciyurt.com), both in the physical building network (UniFi Wi-Fi 6, 5 VLANs, 5651 timestamp) and in the cloud and artificial intelligence layer, with all technical details.

What are the fundamental concepts that must be known in network architecture?

When designing a home office network or the infrastructure of a 150-person multi-story student dormitory, most of the problems encountered are summarized by the complaint "the internet is slow"; however, the mechanisms working in the background are far more specific. Before moving on to the optimization steps, it is necessary to clarify the following five fundamental concepts in order to understand why our interventions work:

1. The MTU and MSS balance: Why does packet size break the connection?

The maximum data size a standard Ethernet frame can carry (MTU - Maximum Transmission Unit) is 1500 bytes. However, in almost all FTTH (Fiber to the Home) GPON infrastructures in Turkey, the connection is established with PPPoE (Point-to-Point Protocol over Ethernet). PPPoE adds an additional 8-byte header to each packet. In this case, the actual maximum packet size that can pass through your line without fragmentation drops to 1500 - 8 = 1492 bytes.

In TCP connections, when the 20-byte IPv4 header and the 20-byte TCP header (a total of 40 bytes) are subtracted from these 1492 bytes, the maximum payload that can be carried at once, that is, the MSS (Maximum Segment Size), becomes 1452 bytes:

Layer / Protocol Header Cost Remaining Maximum Size Description
Standard Ethernet 0 bytes (base) 1500 bytes MTU Default frame limit of the local area network (LAN)
PPPoE (VLAN 35 Fiber) 8 bytes 1492 bytes MTU Upper limit that passes through the ISP backbone without fragmentation
TCP/IPv4 Payload (MSS) 40 bytes (20 IP + 20 TCP) 1452 bytes MSS Net application data carried in a single packet
ICMP Ping Payload (-D) 28 bytes (20 IP + 8 ICMP) 1464 bytes payload Largest ping data that can pass without fragmentation (DF)

If MSS clamping is done incorrectly on your router, two extreme problems arise: If the value remains high, a "PMTU Blackhole" (Path MTU Blackhole) occurs; web pages load halfway, and SSH sessions freeze while producing large output. If the value is mistakenly subtracted twice, unnecessary capacity loss and extra processor load occur on every packet.

2. Bufferbloat and smart queue management (SQM)

When your internet line is idle, a ping to 1.1.1.1 may come out at 18 ms. However, if this latency suddenly jumps to 250-400 ms levels while a device on the same network is downloading a large file or backing up to the cloud, it means you are experiencing bufferbloat.

The deep but unmanaged packet buffers (FIFO queues) of the optical network terminal (ONT) or modems on the ISP side line up packets when the line reaches saturation. Urgent packets of a few bytes, such as video calls, voice communication, or DNS queries, have to wait behind the huge download packets ahead of them. The solution is to catch the line rate on your own firewall just before it fills the ISP buffer (at the 92%-95% level) and split each flow into a separate virtual queue with the RFC 8290 FQ_CoDel (Fair Queuing and Controlled Delay) algorithm.

3. DNS privacy (DoT / DoH) and zero-bypass resolution

Even if you set up a DNS filtering layer on your network that blocks advertising, malware, and phishing domain names, modern operating systems and browsers can easily bypass it:

  • Hard-coded DNS: Some IoT devices, TV boxes, and applications ignore the local DNS address you provide via DHCP and send queries directly to 8.8.8.8:53.
  • Encrypted external DNS (DoT and DoH): "Private DNS" on Android devices (DNS-over-TLS over TCP/UDP 853) or "Secure DNS" in browsers (DNS-over-HTTPS over TCP/UDP 443) completely disables your local DNS server.

For real network security, all outbound traffic on port 53 from the local network should be transparently redirected to the local DNS server at the firewall (NAT Redirect), the DoT port 853 should be rejected (Reject - so the device immediately falls back to standard DNS), and to close browsers' DoH tunnels, the use-application-dns.net canary domain should be answered as NXDOMAIN along with known DoH server lists.

4. VLAN segmentation and broadcast storm control

In a multi-user network, running management computers, staff phones, hundreds of student devices, guests and security cameras on the same flat 192.168.1.0/24 network is a major security weakness. A device on the same subnet can eavesdrop on traffic via ARP spoofing or access the management interface of the security cameras.

With IEEE 802.1Q VLAN tagging, completely isolated virtual networks are set up over a single physical cable and switch. In addition, when there are media devices on the network with both wired and wireless interfaces active at the same time (for example Sonos speakers or wireless bridges), if the RSTP (Rapid Spanning Tree Protocol) priorities on the switch are not set correctly, a packet loop (broadcast storm) occurs within seconds that locks up the entire network.

What security measures do we take when sharing network configurations?

Our fundamental rule is not to compromise operational security (OPSEC) while clearly describing a technical architecture. We apply the following security boundaries both in our own systems and in this article:

  • Masking infrastructure identities: Real external (WAN) IP addresses, ISP BRAS gateway addresses, PPPoE subscriber identities, device MAC addresses, cloud database unique identifiers (UUID) and internal management domain names are never shared as plain text; in the examples, RFC 5737 documentation blocks (203.0.113.0/24, 198.51.100.0/24) and RFC 1918 private network blocks (192.168.1.0/24, 10.10.0.0/16) are used.
  • Zero open ports policy on the external network (WAN): On the WAN interface of our firewalls, no inbound ports are open, including remote access, web management interface, camera monitoring or database services. Remote management is done only through outbound-only, post-quantum cryptography (X25519MLKEM768) supported Cloudflare Zero Trust tunnels and WireGuard, with multi-factor authentication (OTP/TOTP) required.
  • Full isolation of security cameras (CCTV) from the internet: IP cameras and NVR recording devices run on their own private VLAN networks; their outbound internet access is completely blocked at the firewall. This makes it impossible for a potential zero-day vulnerability in camera software to be exploited from outside or for the devices to join a botnet.

1. Local Linux workstation network optimization with Antigravity and the council

The first link in end-to-end optimization was tuning the kernel network stack and wireless connection stability of the Omarchy (Arch Linux) workstation we work on. With the jcode and claude-architect agents in the council, we implemented the following permanent improvements at the /etc/sysctl.d/ and /etc/NetworkManager/ layers:

TCP BBR congestion control and 16 MiB socket buffers

The Linux kernel's default cubic algorithm sharply shrinks the send window even when it sees a single packet loss on the line. Instead, we made Google's BBR algorithm, which models the line's actual bandwidth and round-trip time (RTT), and the FQ (Fair Queuing) packet scheduler permanent with the /etc/sysctl.d/99-network-bbr.conf and 99-omarchy-sysctl.conf files:

# /etc/sysctl.d/99-network-bbr.conf
net.core.default_qdisc = fq
net.ipv4.tcp_congestion_control = bbr
net.core.wmem_max = 16777216
net.core.rmem_max = 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216

# PPPoE ve tünel hatlarında PMTU kara deliklerini önleyen RFC 4821 keşfi
net.ipv4.tcp_mtu_probing = 1

Turning off wireless power-save delays and multicast noise

On modern Wi-Fi 6E and Wi-Fi 7 chips (especially the Intel BE200/BE211 series), NetworkManager's default power-save mode puts the radio layer to sleep when the connection is idle. This causes sudden latency spikes (jitter) of between 20 ms and 300 ms when sending packets. In addition, the LLMNR and mDNS protocols in systemd-resolved both generate unnecessary multicast traffic on the local network and open the door to name poisoning attacks on the local network. We shut down both problems at the root:

# /etc/NetworkManager/conf.d/omarchy-wifi-powersave.conf
[connection]
wifi.powersave = 2

# /etc/systemd/resolved.conf.d/10-disable-multicast.conf
[Resolve]
LLMNR=no
MulticastDNS=no

In addition, we hardened the omarchy-netradar tool we developed to monitor devices and latency on the local network according to HANCORE security standards: The tool can only scan private network blocks (RFC 1918, RFC 3927 and loopback), isolates all its subprocesses in separate process groups (process_group(0)), and prevents argument injection by using the -- flag terminator in external command calls.

2. OPNsense (FreeBSD 15.1) kernel, SQM and DNS optimization

At the heart of the network is OPNsense 26.7.7 (FreeBSD 15.1) running on a 4-core Intel N150 processor, 8 GB RAM and an NVMe SSD. When we examined the existing configuration backup line by line with the concept's 1M+ context gemini-cli and research agents, we identified critical bottlenecks overlooked in standard installations and resolved them one by one.

Hidden capacity loss: fixing the double MSS clamping bug

When we ran validation tests with the DF (Don't Fragment) flag on our Türk Telekom FTTH GPON (VLAN 35 / PPPoE) line, we confirmed that the line supports a full 1492-byte MTU:

# 1464 bayt yük + 20 bayt IP + 8 bayt ICMP = 1492 bayt (%0 kayıp)
ping -D -s 1464 1.1.1.1

# 1465 bayt yük = 1493 bayt -> "frag needed and DF set (MTU 1492)"
ping -D -s 1465 1.1.1.1

However, there is a fairly common but important nuance in OPNsense interface configuration: when you enter 1452 directly in the <mss> field on the WAN interface, OPNsense subtracts the 40 bytes of the TCP/IP header a second time in the background and generates the rule scrub on pppoe1 inet all max-mss 1412 in the pf firewall. This means 40 bytes are wasted in every TCP packet. When we corrected the value on the WAN interface to <mss>1492</mss>, OPNsense itself subtracted 40 bytes and generated the max-mss 1452 value on pf, which is exactly what it should be.

Hardware interrupt (IRQ) isolation and FreeBSD kernel settings

On the 4-core Intel N150 processor, we set up kernel task sharing (CPU Pinning) with a boot hook so that network packets, encryption operations and user-space services do not wait on each other on the same core:

Processor Core Assigned Task and Interrupt (IRQ) Architectural Rationale
CPU 0 NetISR (hybrid, QLimit=4096) + Dummynet FQ_CoDel Uninterrupted operation of packet forwarding and intelligent queue scheduling
CPU 1 re0 (LAN Gigabit Ethernet interrupt) Local network traffic not colliding with WAN interrupts
CPU 2 re1 (WAN GPON VLAN 35 / PPPoE interrupt) Isolating the PPPoE encapsulation load on a single core
CPU 3 User space (unbound, cloudflared, crowdsec, dnsmasq) DNS filtering and tunnel encryption not delaying packet transmission

On the physical Ethernet interfaces, we disabled hardware TSO/LRO and VLAN filtering (-vlanhwtso -tso -lro), which conflicted with software queues; as a result, we reached the values Ierrs=0, Oerrs=0, Idrop=0, Coll=0 (zero packet errors and zero queue drops) in the interface statistics. In addition, to prevent the ZFS file system from consuming all of the memory on the system with 8 GB of RAM, we enabled the vfs.zfs.arc.max = 2147483648 (2 GB) limit and Intel Speed Shift hardware frequency management (dev.hwpstate_intel.{0..3}.epp = 20).

Dummynet FQ_CoDel pipes and Waveform Bufferbloat A+ result

To eliminate bufferbloat on the asymmetric 1000 Mbps / 50 Mbps fiber line, we defined two FQ_CoDel pipes on the OPNsense Traffic Shaper (Dummynet) that fully comply with the RFC 8290 standard:

  • Download Pipe (WAN_Down_930M_FQCoDel): 930 Mbit/s bandwidth, target = 5 ms, interval = 100 ms, quantum = 1492 matching the PPPoE packet size exactly, limit = 4096, flows = 4096 and ECN = enabled.
  • Upload Pipe (WAN_Up_46M_FQCoDel): 46 Mbit/s bandwidth, target = 5 ms, interval = 100 ms, the RFC 8290 low quantum standard (quantum = 300) that prioritizes small ACK/DNS packets on lines below 100 Mbps, limit = 1024, flows = 1024 and NoECN.

The live performance measurements we carried out after the optimization gave the following results:

Measurement Point / Scenario Lowest Average Highest Deviation (Jitter) Packet Loss
ISP First Hop (BRAS - Idle) 1.24 ms 1.64 ms 2.00 ms 0.27 ms 0.0%
Türk Telekom Backbone DNS (dpinger) 4.25 ms 5.31 ms 6.00 ms 0.52 ms 0.0%
Cloudflare Anycast (1.1.1.1) 18.06 ms 18.57 ms 19.00 ms 0.31 ms 0.0%
Google Anycast (8.8.8.8) 25.14 ms 25.94 ms 27.01 ms 0.62 ms 0.0%
Download at Full Load (8 Parallel Streams) — 8.67 ms (+7.04 ms) — — A+ Grade
Upload at Full Load (6 Parallel Streams) — 3.21 ms (-0.56 ms) — — A+ Grade

Unbound DNS (1.26.1): 1.1 GB memory savings and a zero-latency architecture

The existing Unbound configuration had two critical problems: First, because both plain-text Query Forwarding and DNS-over-TLS (DoT) were left enabled at the same time, more than 3,000 queries were falling into timeout (queries_discard_timeout). Second, 11 different overlapping blocklists (2.89 million domain names) were causing Unbound to consume 2.37 GB of RAM on its own.

  1. TLS 1.3 DoT only: We disabled plain-text forwarding; we kept only the cloudflare-dns.com (1.1.1.1:853, 1.0.0.1:853) and dns.google (8.8.8.8:853, 8.8.4.4:853) DoT endpoints.
  2. Targeted DNSBL and 1.1 GB RAM savings: We removed the duplicate lists and enabled HaGeZi Multi PRO Wildcard, Threat Intelligence Feeds, Abuse.ch ThreatFox, and the Pure DoH blocklist with 3,283 servers (doh.txt). We changed the response mode from 0.0.0.0 to NXDOMAIN (nxdomain = 1); this way, client applications get an instant response without waiting for a connection timeout. The total number of unique domain names was fixed at 1,709,008, and Unbound memory usage dropped from 2.37 GB to 1.28 GB (15.8%).
  3. Canary domain and Apple iCloud exemption: To shut down browsers' hidden DoH tunnels, we added the use-application-dns.net canary domain to the blocklist, and to keep the iCloud Private Relay service from breaking on Apple devices, we added mask.icloud.com, mask-h2.icloud.com, and mask-api.icloud.com to the allowlist.
  4. Enforced redirection on PF: On the firewall, we wrote a NAT rdr rule that transparently redirects all TCP/UDP 53 packets leaving the LAN to 127.0.0.1:53, and a filter rule that drops external TCP/UDP 853 (DoT) packets with Reject (TCP RST / ICMP Unreachable) within 3.5 ms.

Why did we choose CrowdSec and Cloudflare Zero Trust instead of Suricata and Zenarmor?

A trap OPNsense users often fall into is installing both Suricata (IPS) and Zenarmor (os-sensei) at the same time on low-power firewalls. In our council analysis, we laid out with concrete data why these two packages should not be used on this hardware:

  • Both engines try to hook FreeBSD's netmap kernel framework on the same interface; this leads to interrupt conflicts and packet drops.
  • Today, when encrypted HTTPS/QUIC traffic exceeds 95%, trying to scan packet payloads without TLS inspection (SSL Inspection) provides very little additional visibility in return for high CPU cost; moreover, Zenarmor's database services alone consume 2 to 4 GB of RAM.

Instead, we built a much more effective pair that never disrupts the kernel packet path (netmap):

  • os-crowdsec (Behavioral IPS and Threat Intelligence): It analyzes firewall and service logs in user space and writes attacker IPs directly to the pf table (crowdsec_blocklists, 1,000,000 table limit); it adds zero latency to the packet path.
  • os-cloudflared (Post-Quantum Zero Trust Tunnel): Without opening even a single port on the WAN, it places the management interface behind Cloudflare Zero Trust Access (authorized email and one-time verification code only) using the QUIC protocol and X25519MLKEM768 post-quantum hybrid key exchange.

3. New setup for Kastamonu İnci Yurt (kastamonuinciyurt.com)

We applied this approach in network and security architecture at enterprise scale in the Kastamonu Özel İnci Yükseköğrenim Kız Öğrenci Yurdu project, at both the physical building network and regulatory layer and the website layer. Since this is an institutional network, I am not including details such as the address plan, port numbers and device locations in this section.

Physical building network: 16 UniFi Wi-Fi 6 access points and a central island architecture

The dormitory building consists of four floors, 100-150 students and 15 staff. When we examined the floor plans, we saw that in the middle of the intermediate floors there was a “central island” (inner rooms and study halls) surrounded by corridors. The classic “let's put one access point at each end of the corridor” approach caused the 5 GHz signal to be completely attenuated by the reinforced-concrete inner room walls.

To solve this, we positioned a total of 16 UniFi Wi-Fi 6 access points using a cellular coverage plan:

  • The existing 6 UniFi U6-Lite units were retained; on top of them, 8 new UniFi U6+ units and, for common areas where students connect intensively at the same time, high-capacity 2 UniFi U6-Pro units were added.
  • All cabling was run down to the central cabinet using 100% Copper Cat6 LSZH (Low Smoke Zero Halogen) cables that emit no smoke and contain no halogens, in accordance with student dormitory fire regulations.

Zero-trust segmentation with 5 separate VLANs and 5651 regulatory compliance

We completely isolated all traffic inside the building from one another with 5 separate VLANs on OPNsense and a managed PoE switch:

  • Staff network: Staff devices receive static addresses mapped to their identities; 5651 records are kept before the Captive Portal.
  • Student network: A Captive Portal with T.C. Identity (NVİ) and FreeRADIUS authentication, a fair per-student speed limit and blocking of access to other networks.
  • Guest network: A visitor network with a speed limit applied and fully isolated from the internal network.
  • Security cameras: IP cameras and the recording device are on a separate network; the cameras' outbound internet access is completely closed, recordings are retained for periods in line with GSB standards and protected by an uninterruptible power supply.
  • Media and IoT: Television boxes and sound systems in common areas; RSTP priority was set on the switch against broadcast loops.

Consolidation on a single device: OPNsense ZFS RAID-1 Mirror architecture

Running a separate firewall, a separate UniFi Cloud Key, a separate 5651 log server and a separate VPN device in the dormitory increased both the number of failure points and energy consumption. Without introducing a virtual machine layer (TrueNAS/Proxmox) in between, we combined 6 critical tasks on a single OPNsense device running directly on a fanless Mini PC with dual NVMe SSDs (ZFS RAID-1 Mirror):

  1. Core Router, Firewall and FQ_CoDel Traffic Shaper.
  2. Built-in UniFi Controller (os-unifi-maxit): Manages 16 Wi-Fi 6 access points and PoE switches without requiring additional devices.
  3. 5651 Student Captive Portal and os-freeradius Authentication Engine.
  4. TÜBİTAK Kamu SM Automatic Time Stamper: Every night it seals that day's IP assignment, MAC and NAT records with a cryptographic time stamp in accordance with Law No. 5651 and stores them for 2 years on dual-disk ZFS RAID-1 mirrored storage (no data loss occurs even if one of the disks fails).
  5. Cloudflare Zero Trust Tunnel (cloudflared): Secure access to dormitory management panels without opening ports to the outside.
  6. WireGuard VPN (os-wireguard): Secure remote access for authorized administrators.

Website

kastamonuinciyurt.com was moved from a static structure to an edge application running on Cloudflare Workers. The management portal, multilingual AI advisor and agent discovery records described in the first version of the article were removed in October 2026; today the site consists of pages introducing the dormitory, ready-made answers to frequently asked questions and the latest Instagram posts.

My reading

  • What is certain: Improving network performance and security does not always require buying more expensive hardware or higher-speed packages. Fixing a 40-byte MSS error that was being double-counted on the PPPoE line, cleaning up conflicting blocklists in Unbound DNS and switching to NXDOMAIN mode, and cutting bufferbloat with FQ_CoDel take the network to a completely different level on the same hardware.
  • What makes the most difference in practice: In institutional buildings (especially multi-storey student dormitories), access point placement done without reading the physical architecture and flat network setups are the biggest source of problems. Moving the cameras to a separate network and cutting their internet access, separating the student and staff networks, and consolidating the entire system in a single OPNsense box with ZFS RAID-1 mirroring dramatically increased both security and ease of maintenance.
  • In summary: Whether you manage your personal Linux workstation and home firewall or set up the network of an institution with 150 people; when correct queue management at the kernel level (BBR + FQ_CoDel), zero-latency DNS, a zero-open-port policy on the external network and the layered oversight of a multi-model council come together, both high speed and enterprise-level security can be achieved at the same time.

References