Prolixium Wiki - User contributions [en]

Prolixium Communications Network

2026-04-04T15:19:05Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', '''Prolixium .NET''', and '''My Hobby Network''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. Some of the PCN nodes are connected via residential data services ([[cable modem]]), while others are located in [[data center|data centers]] have [[Gigabit Ethernet]] (or better) connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of April 4, 2026, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[WireGuard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[daedalus]].prolixium.com on Virtual I/O via [https://tier.net/ Tier.Net]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[T-Mobile USA]] (5G)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[Clover, SC]]: [[trefoil]].prolixium.com on ADSL via [[Spectrum]]
* [[York, SC]]: [[exodus]].prolixium.com on 5G via [[AT&T Mobility]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via Google Fiber
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr (The Constant Company), ARP Networks, Free Range Cloud, and Tier.Net. A Hurricane Electric BGP tunnel is used as backups off excalibur & trident but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], [[daedalus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356
* daedalus: AS397423

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 5 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo
* [[trefoil]] - RPi 5 connected to Spectrum

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:wan.png

2026-04-04T15:18:44Z

Prolixium: Prolixium uploaded a new version of File:wan.png

PCN WAN Architecture

Prolixium Communications Network

2026-04-04T15:18:24Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', '''Prolixium .NET''', and '''My Hobby Network''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. Some of the PCN nodes are connected via residential data services ([[cable modem]]), while others are located in [[data center|data centers]] have [[Gigabit Ethernet]] (or better) connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of March 10, 2024, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[WireGuard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[daedalus]].prolixium.com on Virtual I/O via [https://tier.net/ Tier.Net]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[T-Mobile USA]] (5G)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[Clover, SC]]: [[trefoil]].prolixium.com on ADSL via [[Spectrum]]
* [[York, SC]]: [[exodus]].prolixium.com on 5G via [[AT&T Mobility]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via Google Fiber
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr (The Constant Company), ARP Networks, Free Range Cloud, and Tier.Net. A Hurricane Electric BGP tunnel is used as backups off excalibur & trident but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], [[daedalus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356
* daedalus: AS397423

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 5 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo
* [[trefoil]] - RPi 5 connected to Spectrum

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

Movies seen by Mark Kamichoff

2026-03-29T20:06:22Z

Prolixium:

[[Mark Kamichoff]] recently started keeping track of what movies he's seen.

== 2007 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0465602/ Shoot 'Em Up] || 2007/09/08 22:00 [[EDT]] || Entertaining || [http://en.wikipedia.org/wiki/Regal_Entertainment_Group Regal Entertainment Group]
|-
|[http://www.imdb.com/title/tt0431197/ The Kingdom] || 2007/09/28 17:35 EDT || Entertaining || [http://en.wikipedia.org/wiki/AMC_Theatres AMC Theatres]
|-
|[http://www.imdb.com/title/tt0465538/ Michael Clayton] || 2007/10/19 20:05 EDT || [[Good]] || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0765429/ American Gangster] || 2007/11/09 19:40 [[EST]] || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0480249/ I Am Legend] || 2007/12/15 21:50 EST || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0465234/ National Treasure: Book of Secrets] || 2007/12/28 20:25 EST || Boring || Regal Entertainment Group
|-
|}

== 2008 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0880578/ Untraceable] || 2008/02/09 20:00 EST || Unsettling || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1060277/ Cloverfield] || 2008/02/14 19:05 EST || Wow || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443274/ Vantage Point] || 2008/02/22 20:15 EST || Predictable || AMC Theatres
|-
|[http://www.imdb.com/title/tt0478087/ 21] || 2008/03/29 19:50 EST || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1091617/ Expelled: No Intelligence Allowed] || 2008/04/23 19:25 EDT || Revealing || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0411061/ 88 Minutes] || 2008/04/25 19:30 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0371746/ Iron Man] || 2008/05/07 20:00 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0367882/ Indiana Jones and the Kingdom of the Crystal Skull] || 2008/05/23 18:45 EDT || Entertaining || AMC Theatres
|-
|[http://www.imdb.com/title/tt0493464/ Wanted] || 2008/06/27 19:55 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0468569/ The Dark Night] || 2008/07/18 21:00 EDT || Awesome (but too long) || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443701/ The X Files: I Want to Believe] || 2008/07/25 19:55 EDT || Blasphemous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0988047/ Traitor] || 2008/09/01 19:50 EDT || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1034331/ Righteous Kill] || 2008/09/12 20:30 EDT || Interesting || AMC Theatres
|-
|[http://www.imdb.com/title/tt0887883/ Burn After Reading] || 2008/09/19 19:20 EDT || Hilarious || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0830515/ Quantum of Solace] || 2008/11/15 15:55 EST || [http://www.imdb.com/title/tt0381061/ Casino Royale] was better || Regal Entertainment Group
|-
|}

== 2009 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0421715/ The Curious Case of Benjamin Button] || 2009/01/09 22:00 EST || Excellent || [http://www.mezcharlotte.com/ MEZ]
|-
|[http://www.imdb.com/title/tt1114740/ Paul Blart: Mall Cop] || 2009/02/06 19:05 EST || Painful, yet humorous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0963178/ The International] || 2009/03/14 16:00 EDT || Banks are evil? || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0473705/ State of Play] || 2009/04/24 19:45 EDT || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0458525/ X-Men Origins: Wolverine] || 2009/04/30 21:20 EDT || Action-packed || [http://www.ayrsleycinemas.com/ Ayrsley Cinemas]
|-
|[http://www.imdb.com/title/tt0796366/ Star Trek] || 2009/05/07 19:45 EDT || [http://www.prolixium.com/mynews?id=839 Good] || Ayrsley Cinemas
|-
|Star Trek || 2009/05/08 16:50 EDT || Good || AMC Theatres
|-
|Star Trek || 2009/05/09 21:30 EDT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt0808151/ Angels & Demons] || 2009/05/15 19:30 EDT || Book was better, except for the end || Regal Entertainment Group
|-
|Star Trek ([[IMAX]]) || 2009/05/20 19:15 EDT || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1049413/ Up] || 2009/06/12 19:00 EDT || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0417741/ Harry Potter and the Half-Blood Prince] || 2009/08/01 16:30 EDT || Blah || AMC Theatres
|-
|[http://www.imdb.com/title/tt0361748/ Inglorious Basterds] || 2009/08/23 16:00 EDT || Violent || Ayrsley Cinemas
|-
|[http://www.imdb.com/title/tt1136608/ District 9] || 2009/08/29 17:40 PDT || Surprising || [http://www.pacifictheatres.com/ Pacific Theatres]
|-
|[http://www.imdb.com/title/tt1190080/ 2012] || 2009/11/24 12:05 EST || Thrilling || AMC Theatres
|-
|[http://www.imdb.com/title/tt0499549/ Avatar] [[3D]] || 2009/12/26 23:30 EST || Awesome || AMC Theatres
|-
|}

== 2010 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1228705/ Iron Man 2] || 2010/05/07 22:00 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0944835/ Salt] || 2010/07/24 13:50 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1323594/ Despicable Me] 3D || 2010/07/30 19:10 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1375666/ Inception] || 2010/08/07 15:25 EDT || Intriguing || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1001526/ Megamind] 3D || 2010/11/27 17:50 EST || Quite good || AMC Theatres
|-
|[http://www.imdb.com/title/tt1104001/ Tron: Legacy] 3D || 2010/12/17 16:00 EST || Awesome || AMC Theatres
|-
|[http://www.imdb.com/title/tt0980970/ The Chronicles of Narnia: The Voyage of the Dawn Treader] || 2010/12/23 18:50 EST || Not bad || AMC Theatres
|}

== 2011 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0970866/ Little Fockers] || 2011/01/02 16:30 EST || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0993842/ Hanna] || 2011/05/07 19:20 EDT || Strange || AMC Theatres
|-
|[http://www.imdb.com/title/tt0458339/ Captain America: The First Avenger] || 2011/08/07 14:35 EDT || Exciting || AMC Theatres
|-
|[http://www.imdb.com/title/tt1509767/ The Three Musketeers] || 2011/11/04 22:25 EDT || Fun || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1568911/ War Horse] || 2011/12/29 15:35 EST || Meh || Regal Entertainment Group
|}

== 2012 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1229238/ Mission: Impossible - Ghost Protocol] (IMAX) || 2012/01/06 22:10 EST || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0477302/ Extremely Loud and Incredibly Close] || 2012/01/20 21:40 EST || Well done || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1591479/ Act of Valor] || 2012/03/02 22:50 EST || Powerful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1232829/ 21 Jump Street] || 2012/03/30 22:00 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0848228/ The Avengers] || 2012/05/05 21:30 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1409024/ Men in Black III] || 2012/06/09 16:10 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2215285/ Madea's Witness Protection] || 2012/06/30 16:10 EDT || Humorous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1345836/ The Dark Night Rises] || 2012/07/27 21:00 EDT || Excellent || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1790886/ The Campaign] || 2012/08/18 16:50 EDT || Funny, but over the line || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1276104/ Looper] || 2012/10/06 19:30 EDT || Strange || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1074638/ Skyfall] || 2012/11/10 17:20 EST || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443272/ Lincoln] || 2012/11/22 19:30 EST || Good || AMC Theatres
|}

== 2013 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1707386/ Les Miserables] || 2013/01/01 16:00 EST || Masterpiece || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1790885/ Zero Dark Thirty] || 2013/01/26 16:35 EST || Dramatic || AMC Theatres
|-
|[http://www.imdb.com/title/tt1606378/ A Good Day to Die Hard] || 2013/02/23 17:40 EST || Explosive || AMC Theatres
|-
|[http://www.imdb.com/title/tt1623205/ Oz the Great and Powerful] || 2013/03/23 17:05 EDT || Childish || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1483013/ Oblivion] || 2013/04/19 22:20 EDT || Beautiful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1300854/ Iron Man 3] || 2013/05/10 19:15 EDT || Exciting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1408101/ Star Trek Into Darkness] (IMAX 3D) || 2013/05/18 12:50 EDT || Enjoyable || Regal Entertainment Group
|-
|Star Trek Into Darkness || 2013/06/15 19:00 EDT || Enjoyable || AMC Theatres
|-
|Star Trek Into Darkness || 2013/06/30 15:20 EDT || Enjoyable || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0770828/ Man of Steel] || 2013/07/04 17:30 EDT || Gratuitous Destruction || AMC Theatres
|-
|[http://www.imdb.com/title/tt1723121/ We're the Millers] || 2013/08/17 20:10 EDT || Funny, but vulgar || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2357129/ Jobs] || 2013/08/19 19:20 EDT || Inspiring || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1411250/ Riddick] || 2013/09/10 19:40 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1837703/ The Fifth Estate] || 2013/10/23 19:30 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1535109/ Captain Phillips] || 2013/10/26 21:15 EDT || Suspenseful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1981115/ Thor: The Dark World] || 2013/11/10 19:00 EST || Mostly Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt3063516/ Jackass Presents: Bad Grandpa] || 2013/11/23 19:00 EST || Funny || AMC Theatres
|}

== 2014 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1205537/ Jack Ryan: Shadow Recruit] || 2014/01/25 17:20 EST || Suspenseful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1418377/ I, Frankenstein] || 2014/02/15 20:00 EST || Strange, but good || AMC Theatres
|-
|[http://www.imdb.com/title/tt2872732/ Lucy] || 2014/08/02 14:10 PDT || Disappointing || AMC Theatres
|-
|[http://www.imdb.com/title/tt1790864/ The Maze Runner] || 2014/10/11 15:25 PDT || Great || AMC Theatres
|-
|[http://www.imdb.com/title/tt0816692/ Interstellar] || 2014/11/09 14:10 PST || Great || AMC Theatres
|-
|[http://www.imdb.com/title/tt1809398/ Unbroken] || 2014/12/25 19:40 EST || Long, Unsettling || AMC Theatres
|}

== 2015 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt2395427/ Avengers: Age of Ultron] || 2015/05/15 17:30 PDT || Silly || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0369610/ Jurassic World] || 2015/06/20 13:05 PDT || Greedy || [https://en.wikipedia.org/wiki/Cinemark_Theatres Cinemark Theatres]
|-
|[http://www.imdb.com/title/tt4046784/ Maze Runner: The Scorch Trials] || 2015/09/19 16:15 PDT || Didn't match the book || Cinemark Theatres
|-
|[http://www.imdb.com/title/tt2279339/ Love the Coopers] || 2015/11/25 12:40 EST || Alright || Regal Cinemas
|-
|[http://www.imdb.com/title/tt2488496/ Star Wars: The Force Awakens] || 2015/12/29 13:45 EST || Great || Ayrsley Cinemas
|}

== 2016 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt2277860/ Finding Dory] || 2016/06/25 16:20 PDT || Fun || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2709768/ The Secret Life of Pets] || 2016/07/09 14:00 PDT || Fun || AMC Theatres
|-
|[http://www.imdb.com/title/tt2660888/ Star Trek Beyond] || 2016/07/31 13:30 PDT || Too Much Action || AMC Theatres
|-
|[http://www.imdb.com/title/tt2387499/ Keeping Up with the Joneses] || 2016/10/29 11:50 PT || Fun || AMC Theatres
|-
|[http://www.imdb.com/title/tt2543164/ Arrival] || 2016/11/19 16:05 PT || Awesome || AMC Theatres
|-
|[http://www.imdb.com/title/tt3183660/ Fantastic Beasts and Where to Find Them] || 2016/12/04 13:30 PT || Alright || AMC Theatres
|}

== 2017 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt3748528/ Rogue One: A Star Wars Story] || 2017/01/02 11:30 PT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt1219827/ Ghost in the Shell] || 2017/04/08 10:20 PT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt3896198/ Guardians of the Galaxy Vol. 2] || 2017/06/11 14:15 PT || Excellent || AMC Theatres
|-
|[http://www.imdb.com/title/tt3469046/ Despicable Me 3] || 2017/07/04 14:45 PT || Mildly Funny || AMC Theatres
|-
|[http://www.imdb.com/title/tt2239822/ Valerian and the City of a Thousand Planets] || 2017/07/29 17:10 PT || Decent || AMC Theatres
|-
|[http://www.imdb.com/title/tt1856101/ Blade Runner 2049] || 2017/10/22 1510 PT || Need to rewatch the original || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt3501632/ Thor: Ragnarok] || 2017/12/02 1540 PT || Funny || AMC Theatres
|-
|[http://www.imdb.com/title/tt2527336/ Star Wars: The Last Jedi] || 2017/12/18 1215 PT || Entertaining || AMC Theatres
|-
|Star Wars: The Last Jedi || 2017/12/25 1930 ET || Entertaining || AMC Theatres
|}

== 2018 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1825683/ Black Panther] || 2018-03-17 1045 PT || Lived up to the hype || AMC Theatres
|-
|[https://www.imdb.com/title/tt4154756/ Avengers: Infinity War] || 2018-04-27 1645 PT || Wow || AMC Theatres
|-
|[https://www.imdb.com/title/tt4123430/ Fantastic Beasts: The Crimes of Grindelwald] || 2018-12-09 1500 PST || Meh || Regal Entertainment Group
|}

== 2019 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt2527338/ Star Wars: Episode IX - The Rise of Skywalker] || 2019-12-25 1515 PST || Decent || AMC Theatres
|}

== 2021 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt1160419/ Dune] || 2021-10-22 2040 PDT || Good || Regal Entertainment Group
|}

== 2022 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt1745960/ Top Gun: Maverick] || 2022-06-25 1615 EDT || Great || [https://en.wikipedia.org/wiki/Alamo_Drafthouse_Cinema Alamo Drafhouse Cinema]
|}

== 2023 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt6791350/ Guardians of the Galaxy Vol. 3] || 2023-05-06 1615 EDT || Good || Alamo Drafhouse Cinema
|-
|[https://www.imdb.com/title/tt1462764/ Indiana Jones and the Dial of Destiny] || 2023-07-04 1100 EDT || Good || Alamo Drafhouse Cinema
|-
|[https://www.imdb.com/title/tt15398776/ Oppenheimer] || 2023-07-22 0940 EDT || Monumental || Regal Entertainment Group
|}

== 2024 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt1856080/ The Boys in the Boat] || 2024-01-01 1535 EST || Suspenseful || Regal Entertainment Group
|}

== 2026 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt12042730/ Project Hail Mary] || 2026-03-29 1130 EDT || Excellent || [https://www.cmxcinemas.com/ CMX Cinemas]
|}

2024-05-12T22:45:33Z

Prolixium: /* SmokePing */

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', '''Prolixium .NET''', and '''My Hobby Network''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. Some of the PCN nodes are connected via residential data services ([[cable modem]]), while others are located in [[data center|data centers]] have [[Gigabit Ethernet]] (or better) connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of March 10, 2024, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[WireGuard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[daedalus]].prolixium.com on Virtual I/O via [https://tier.net/ Tier.Net]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[Clover, SC]]: [[trefoil]].prolixium.com on ADSL via [[Spectrum]]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr (The Constant Company), ARP Networks, Free Range Cloud, and Tier.Net. A Hurricane Electric BGP tunnel is used as backups off excalibur & trident but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], [[daedalus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356
* daedalus: AS397423

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo
* [[trefoil]] - RPi 5 connected to Spectrum

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:pcn-world.png

2024-05-12T22:37:57Z

Prolixium: Prolixium uploaded a new version of File:pcn-world.png

File:wan.png

2024-05-12T22:34:56Z

Prolixium: Prolixium uploaded a new version of File:wan.png

PCN WAN Architecture

Prolixium Communications Network

2024-05-12T22:18:21Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', '''Prolixium .NET''', and '''My Hobby Network''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. Some of the PCN nodes are connected via residential data services ([[cable modem]]), while others are located in [[data center|data centers]] have [[Gigabit Ethernet]] (or better) connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of March 10, 2024, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[WireGuard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[daedalus]].prolixium.com on Virtual I/O via [https://tier.net/ Tier.Net]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[Clover, SC]]: [[trefoil]].prolixium.com on ADSL via [[Spectrum]]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr (The Constant Company), ARP Networks, Free Range Cloud, and Tier.Net. A Hurricane Electric BGP tunnel is used as backups off excalibur & trident but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], [[daedalus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356
* daedalus: AS397423

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:bgpnet.png

2024-05-12T22:17:20Z

Prolixium: Prolixium uploaded a new version of File:bgpnet.png

BGP Confederations on [[PCN]]

Prolixium Communications Network

2024-03-10T20:17:33Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', '''Prolixium .NET''', and '''My Hobby Network''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. Some of the PCN nodes are connected via residential data services ([[cable modem]]), while others are located in [[data center|data centers]] have [[Gigabit Ethernet]] (or better) connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of March 10, 2024, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[WireGuard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[daedalus]].prolixium.com on Virtual I/O via [https://tier.net/ Tier.Net]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr (The Constant Company), ARP Networks, Free Range Cloud, and Tier.Net. A Hurricane Electric BGP tunnel is used as backups off excalibur & trident but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], [[daedalus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356
* daedalus: AS397423

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:pcn-world2-transit.png

2024-03-10T13:38:31Z

Prolixium: Prolixium uploaded a new version of File:pcn-world2-transit.png

File:pcn-world.png

2024-03-10T13:37:47Z

Prolixium: Prolixium uploaded a new version of File:pcn-world.png

Prolixium Communications Network

2024-03-03T00:18:19Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', and '''Prolixium .NET''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. The majority of the PCN nodes are connected via residential data services ([[cable modem]]), while some located in [[data center|data centers]] have [[Gigabit Ethernet]] connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of February 2, 2022, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[Wireguard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]
** [[excalibur]].prolixium.com on Virtual I/O via [https://www.vultr.com/ Vultr]
** [[dax]].prolixium.com on Virtual I/O via Vultr
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via Vultr
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr, Choopa (The Constant Company), ARP Networks, and Free Range Cloud. A Hurricane Electric BGP tunnel is used as backups in LAX and EWR2 but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:pcn-world2-transit.png

2024-03-03T00:15:34Z

Prolixium: Prolixium uploaded a new version of File:pcn-world2-transit.png

File:bgpnet-transit.png

2024-03-03T00:13:34Z

Prolixium: Prolixium uploaded a new version of File:bgpnet-transit.png

Border Transit Network

File:smokeping.png

2024-02-14T01:36:57Z

Prolixium: Prolixium uploaded a new version of File:smokeping.png

PCN SmokePing

File:wan.png

2024-02-14T01:36:37Z

Prolixium: Prolixium uploaded a new version of File:wan.png

PCN WAN Architecture

Prolixium Communications Network

2024-02-14T01:26:35Z

Prolixium:

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', and '''Prolixium .NET''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. The majority of the PCN nodes are connected via residential data services ([[cable modem]]), while some located in [[data center|data centers]] have [[Gigabit Ethernet]] connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of February 2, 2022, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[Wireguard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]: [[excalibur]].prolixium.com on Gigabit Ethernet via [[Choopa]]
** [[dax]].prolixium.com
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via [[Vultr]]
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Arlington, VA]]: [[merlin]].prolixium.com on Ethernet via Comcast Business / Zayo
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr, Choopa (The Constant Company), ARP Networks, and Free Range Cloud. A Hurricane Electric BGP tunnel is used as backups in LAX and EWR2 but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 180/30.189.9.69.in-addr.arpa., 232/29.186.9.69.in-addr.arpa, 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Google Fiber
* [[merlin]] - RPi 3 B connected to Comcast Business / Zayo

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:charlotte.png

2024-01-09T01:19:50Z

Prolixium: Prolixium uploaded a new version of File:charlotte.png

Charlotte LAN Environment

Prolixium Communications Network

2024-01-09T01:18:55Z

Prolixium: /* Current State */

[[File:pcn.png|thumb|280px|Prolixium Communications Network Logo]]The Prolixium Communications Network (known also as '''PCN''', '''mynet''', '''My Network''', and '''Prolixium .NET''') is a collection of small, geographically disperse, computer networks that provide [[IPv4]] and [[IPv6]], [[VPN]], and [[VoIP]] services to the [[Kamichoff]] family. Owned and operated solely by [[Mark Kamichoff]], PCN often serves as a testbed for various network experiments. The majority of the PCN nodes are connected via residential data services ([[cable modem]]), while some located in [[data center|data centers]] have [[Gigabit Ethernet]] connections to the [[Internet]].

== Current State ==

=== Overview ===

[[file:wan.png|thumb|PCN WAN Architecture]][[file:pcn-world.png|thumb|PCN World Map]]As of February 2, 2022, PCN is composed of several networks in the [[United States]] and across the globe, connected via [[OpenVPN]] and [[Wireguard]] with the IPv6 backbone connected via [[6in4]] tunnels:

* [[North Brunswick, NJ]]: [[nat]].prolixium.com on [[FTTH]] via [[Verizon FiOS]]
* [[Piscataway, NJ]]: [[excalibur]].prolixium.com on Gigabit Ethernet via [[Choopa]]
** [[dax]].prolixium.com
* [[Toronto, Canada]]: [[tiny]].prolixium.com on Virtual I/O via [http://atlantic.net/ atlantic.net]
* [[Dallas, TX]]: [[nox]].prolixium.com on Virtual I/O via [http://www.linode.com/ Linode]
* Dallas, TX: [[concorde]].prolixium.com on Virtual I/O via [[Vultr]]
* [[Ashburn, VA]]: [[pegasus]].prolixium.com on Virtual I/O via [https://freerangecloud.com/ Free Range Cloud]
* Ashburn, VA: [[matrix]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA: [[elise]].prolixium.com on Virtual I/O via [https://cloud.oracle.com/ Oracle Cloud]
* Ashburn, VA
** [[discovery]].prolixium.com via [[Verizon FiOS]]
** [[sprint]].prolixium.com via [[Verizon Wireless]] (LTE)
* [[Seattle, WA]]: [[orca]].prolixium.com on Virtual I/O via Vultr
* Seattle, WA: [[interstellar]].prolixium.com on Virtual I/O via Vultr
* [[Sarasota, FL]]: [[scimitar]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Los Angeles, CA]]: [[trident]].prolixium.com Virtual I/O via [http://www.arpnetworks.com/ ARP Networks]
* [[York, SC]]: [[exodus]].prolixium.com on ADSL via [[AT&T]]
* [[Austin, TX]]: [[photonic]].prolixium.com on FTTH via Google Fiber
* [[Charlotte, NC]]: [[storm]].prolixium.com on FTTH via AT&T
* [[Agawam, MA]]: [[galactica]].prolixium.com on DOCSIS via Comcast Xfinity
* [[Amsterdam, Netherlands]]: [[firefly]].prolixium.com on Virtual I/O via [http://www.digitalocean.com/ DigitalOcean]
* [[Singapore]]: [[centauri]].prolixium.com on Virtual I/O via [http://ec2.amazon.com/ Amazon EC2]

Each site has multiple OpenVPN tunnels to other locations supporting both IPv4 and IPv6. The network is primarily powered by [[Free Range Routing]] (FRR) with some sites using [[BIRD]].

=== Routing ===

The routing infrastructure consists of several autonomous systems, taken from the IANA-allocated private range: 64512 through 65534. Each site runs IBGP, possibly with a route reflector, and its own [[IGP]] for local next-hop resolution. EBGP is used between sites and peering connections. IPv4 Internet connectivity for each site is achieved by advertisement of default routes from boxes performing NAT. The [[Prolixium Communications Network#Lab|lab]] is connected to [[starfire]] (core router) in Ashburn, VA. The PCN used to use one large OSPF area with no EGP. It was converted to a [[BGP]] confederation setup, which was a bad idea (but educational!), then reconverted to its current state.

[[file:bgpnet.png|280px|BGP on PCN]]

=== IPv6 Connectivity ===

IPv6 connectivity is provided by four (5) direct connections to Vultr, Choopa (The Constant Company), ARP Networks, and Free Range Cloud. A Hurricane Electric BGP tunnel is used as backups in LAX and EWR2 but is depreferenced. The border transit network piece of the PCN provides this connectivity.

IPv6 addressing is out of 2620:6:2000::/44, which is a direct allocation from ARIN.

==== Border Transit Network ====

The border transit network operates in AS395460 and consists of [[excalibur]], [[trident]], [[orca]], [[pegasus]], and [[concorde]]. Connectivity is provided by the following transit peers:

* trident: AS25795 and AS6939
* excalibur: AS20473 and AS6939
* orca: AS20473
* concorde: AS20473
* pegasus: AS53356

This network injects a default route into the rest of the PCN, which can be referred to PEN (Prolixium Enterprise Network). The border network receives a full table from all transits and advertises 2620:6:2000::/44 out each peer along with some sites advertising /48 specifics for networks that are nearby.

Hurricane Electric (AS6939) is only used as backup because it is a tunneled connection and is suspected to be throttled.

[[file:bgpnet-transit.png|280px|Border Transit Network]]

[[file:pcn-world2-transit.png|280px|Border Transit Network Map]]

The following hosts do not default route to the border transit network and use their own native IPv6 connectivity:

* centauri
* firefly
* storm

The following hosts may have IPv6 connectivity but it's not currently enabled (at time of writing):

* exodus
* galactica
* photonic

=== DNS ===

[[DNS]] is done with two views: internal and external. PCN has two external nameservers, and four internal ones, all which perform zone transfers from the master nameserver, ns3.antiderivative.net. antiderivative.net is used for all NS records, as well as glue records at the GTLD servers. The internal nameservers are ns{1-4} and external ones are ns{2,3}. Each zone has two views, internal and external, and a common file that is included in both views (SOA, etc.). The zones include the following:

* Internal view, answering to 10/8, 172.16/12, and 192.168/16 addresses
** 3.10.in-addr.arpa. and 3.16.172.in-addr.arpa. reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s internal A/CNAME records
* External view, answering to everything !RFC1918
** prolixium.com, prolixium.net, antiderivative.net, etc.'s external A/CNAME records
* Common information, answering for all hosts
** 180/30.189.9.69.in-addr.arpa., 232/29.186.9.69.in-addr.arpa, 0.0.0.2.6.0.0.0.0.2.6.2.ip6.arpa., and other reverse zones
** prolixium.com, prolixium.net, antiderivative.net, etc.'s common MX records

Previously, the Xicada DNS Service (developed by Mark Kamichoff) kept track of all the forward delegations as well as IPv4 reverse delegations on Xicada. The administrator of each node enumerated their zones into a web form, and then configured their DNS server to pull down a forwarders definition for all Xicada zones. It supported BIND and djbdns, but also outputted a CSV file if someone decided to use another DNS server. It was originally intended that each DNS server should pull down a fesh copy of the forwarders definition file nightly, but there were really no rules.

Mark Kamichoff has a policy on his network to have DNS entries (includes A, AAAA, and PTR) for each and every active IP address. If a host is offline, the DNS records should be immediately expunged. This precludes the requirement of a host management system or a collection of poorly-maintained spreadsheets. If an IP is needed, the PTR should be checked. All [[DHCP]]-assigned IP addresses are created via {side ID}-{lastoctet}.prolixium.com. Again, no confusion. DNS itself is a database, so why not use it?

All transit links on PCN are addressed using the prolixium.net domain. The format is {unit/VLAN}.{interface}.{host}.prolixium.net. For example, the xl1 interface on starfire would be: xl1.starfire.prolixium.net. There is a collection of DNS entries for every IPv4 and IPv6 transit link. There is not one hop in my network which has no PTR record (or a PTR record w/out a corresponding A or AAAA record). Each router has a loopback interface with IPv4 and IPv6 addresses (if supported).

=== Ashburn-Specific Setup ===

[[file:charlotte.png|thumb|Ashburn LAN]]The network setup in Ashburn (formerly Seattle, WA and Charlotte, NC) is slightly different from the other sites, where there is a single router with a dynamic address. In the Ashburn location there are two ISPs and they're terminated in separate LXC instances (all with VPNs to at least one of interstellar, nox, dax, or elise - the "enterprise" network):

* discovery (on evolution) - Verizon FiOS
* sprint (on evolution) - Verizon Wireless (LTE)

starfire and evolution are the two core routers with multiple Gigabit Ethernet interfaces. The current routing setup is as follows:

* IPv6 (Internet & internal) inbound & outbound traffic traverses discovery (Verizon FiOS) via VPN
* IPv4 Internet inbound & outbound traffic traverses discovery (Verizon FiOS) via NAT
* All LXCs above advertise an IPv4 default route into OSPFv2
* LOCAL_PREF and AS_PATH prepending influence the traffic flow

In the case of backup, discovery is replaced with the LXC sprint.

In the past, NetFlow was used on atlantis, which was depicted in the drawing below:

[[file:netflow.png|280px|PCN NetFlow Setup]]

The NetFlow collector ran [http://www.ntop.org/ ntop], but this was uninstalled due to instability.

=== Printing ===

The whole printing/CUPS/lpd setup is mostly an annoyance. Most people would want to run CUPS on every Unix client on the network. Mark Kamichoff believes it's better to have a lightweight client send a [http://en.wikipedia.org/wiki/PostScript PostScript] file via lpd to a CUPS server rather than sending a huge RAW raster stream across the network and have both the client and server do print processing. See the diagram to the bottom:

[[file:printing.png|280px|PCN Printing Setup]]

=== SmokePing ===

For monitoring, PCN uses a combination of Nagios, SmokePing, and [[MRTG]]. The SmokePing setup itself is a combination of slaves and masters, both IPv4 and IPv6.

[[file:smokeping.png|280px|SmokePing]]

[[nox]] is the master for a few slaves:

* [[tiny]] - VPS connected to atlantic.net
* [[storm]] - RPi 3 connected to AT&T Fiber
* [[exodus]] - RPi 3 connected to AT&T DSL
* [[galactica]] - RPi 4 B connected to Comcast Xfinity
* [[photonic]] - RPi 4 B connected to Charter Spectrum

== History ==

<div class="mw-collapsible mw-collapsed">History is hidden by default. Click '''expand''' to see it.<div class="mw-collapsible-content">''Warning: This entire section is written in the first-person ([[Mark Kamichoff|Mark Kamichoff's]]) point of view''

=== Beginnings ===

After joining the [[http://xicada.sf.net Xicada] network back at [[RPI]], I decided to continue linking all of my networks and sites together via various VPN technologies. At first, the network was just a simple VPN between my network at home and a few computers in my dorm room at RPI. The connection tunnelled through RPI's firewall like a knife through warm butter, using OpenVPN's [[UDP]] encapsulation mode. Actually, a site to site UDP tunnel was the only thing OpenVPN offered, back then. My router at RPI was a blazing-fast [[Pentium]] 166MHz box running [[Debian GNU/Linux]]. At that point, my Xicada tunnels were terminated on another box I found in the trash, an old AMD K6-300, which eventually ran FreeBSD 4.

The network quickly started expanding, and I was able to move the K6-300 box (starfire) into the ACM's lab, which was given a 100mbit link, in the basement of the DCC. At this point in time, my network had three sites: home, the lab, and my dorm room. Since I didn't stick around RPI during most summers, I reterminated the Xicada links on starfire, since it sported a more permanent link.

Shortly after starfire was moved to the lab, I started toying with IPv6, and acquired a tunnel via Freenet6 (now Hexago, since they're actually trying to sell products, or something). RPI's firewall wouldn't allow IP protocol 41 through the firewall, and my attempts at getting this opened up for my IP failed. So, I terminated the IPv6 tunnel on my box at home, which sat on Optimum Online. Freenet6 gave me a /48 block out of the 3ffe::/16 6bone space, and I started distributing /64's out to all of my LAN segments. I started running Zebra's OSPFv3 daemon, and realized it was buggy as all get out. It mostly worked, though. Since Freenet6 gave me an ip6.int. delegation, I spent some time applying tons of patches to djbdns, my DNS server of choice, back then. After tons of patching, I got IPv6 support, which was fairly neat at the time. What did I use this new-found IPv6 connectivity for? IRC and web site hosting. www.prolixium.com has had an AAAA record since at least 2003.

Sometime in 2003 (I forget when), I moved my IPv6 tunnel to BTExact, British Telecom's free tunnel broker that actually gave out non-6bone /48's and ip6.arpa. DNS delegations. I quickly moved to them, and enjoyed quicker speeds than Freenet6 for about a year. Of course, after a year, my parents had a power outage at home, and my server lost the IP it had with OOL for the past two years. BTExact, at that time, had frozen their tunnel broker service, and didn't allow any modifications or new tunnels to be created. I went back to Freenet6, who had changed to 2001::/16 space.

After leaving RPI, and getting a job, I decided to purchase a dedicated server from SagoNet. I extended my network down to Tampa, FL, where the server was located.

Fast-forwarding to the present day, I currently have six sites, and native IPv6 from Voxel dot Net. Almost every host on the network is IPv6-aware, and the IPv6 connectivity is controlled completely by pf.

Xicada connectivity at this point has been terminated, due to lack of interest.

=== [[VLAN]] Conversion (Laundry Room Data Center) ===

[[file:vlan.png|thumb|VLAN Setup]]I'm lucky to have CAT5(e?) cabled to every room in my condo, all aggregated in the [[laundry room]], I figured it was time to deploy a couple different VLANs on my network. Initially, I just had a dumb switch connecting all of the various ports in different rooms together. Since that was too simple of a solution, I picked up a Cisco 2940 switch on [http://www.ebay.com/ eBay], and setup a 1Gbit trunk between starfire and the laundry room. I setup 4x VLANs:

* 2: Various wall jacks
* 3: Media center link (connected to kamikaze)
* 4: Linksys link (connected to mercury)
* 5: Lab link (connected to hysteresis)

I ended up throwing some other gear in the laundry room along with the switch, and ended up moving my lab (3.0) there.

=== BGP (Confederations) Conversion ===

==== History ====

Starting with the Xicada project, my network was one big OSPF backbone area. Entirely flat, except for some route redistribution for the lab connection. When I added OSPFv3 for IPv6 reachability, it was no different - one big area: no stub areas, no frills. It worked, but was boring, and didn't provide the flexibility required if I wanted to start redirecting Internet traffic.

After reading up on BGP, I realized I could make my network 1000% more complex, while gaining some real-world experience. Sounds like a plan, huh?
Preparation and Design

Due to some Quagga instability issues, I originally tested out some alternate BGP/OSPF implementations, including XORP. Unfortunately, none of them fit the bill, and XORP, although promising, was horribly unstable and appeared to suffer from configuration file parsing issues, more than anything else. So I decided to stick with Quagga. I also decided to keep two separate BGP connections, one for IPv4 and one for IPv6 (so I didn't run into any nasty next-hop accessibility problems).

One of the goals of the redesign was to eliminate the large network-wide IGP process and break down each site into sub-ASes, using BGP confederations and route reflectors. This required a partial mesh of CBGP (confederation BGP - like EBGP, but more attributes are retained) between all the sites, to take advantage of the tunnels. Unfortunately, this meant that I had to renumber all of my IPv6 tunnels, since they were all /128's. Not a big deal. I didn't want to do this with the IPv4 (OpenVPN) tunnels, since the documentation strongly recommended against the use of anything other than a 32-bit netmask. This required the use of the ebgp-multihop command, since according to most [E]BGP implementations, /32's or /128's connecting to each other is not classified as 'directly connected' for some reason. (doesn't make sense to me, since even a TTL of 1 should theoretically allow communication to succeed)

At each site, I wanted to run IBGP internally, and designate one box to be the route reflector, in order to loosen the IBGP full-mesh requirement. Some of the OpenWrt devices did not have loopbacks at the time, so I needed to shuffle around some addresses and fix this.

I'd still run an IGP internal to each site (not nox or dax, since they are only one router), and advertise a default route via OSPFv2 within the site, for Internet access. I could also advertise default routes from two different routers within a site, for redundancy and failover Internet access.

So, here's some of the tasks I performed prior to making any routing changes:

# Add loopbacks to all routers
# Redo all IPv6 tunnel interfaces, converted to /126's to avoid subnet-router anycast issues
# Redo tunnel naming standards (was too long before)

==== IPv6 Migration ====

I figured, since on most platforms, IGP routes take precedence over BGP routes, I could add all the peering relationships and get everything setup without skipping a beat. Quagga's zebra process wouldn't insert or remove anything from the FIB (the kernel routing table). Then I could remove OSPFv3 from all the WAN links, and zebra would just shuffle around the routes, but reachability would come back within a few minutes, maybe?

So I started building the BGP neighbors, and quickly ran into a problem. For some reason, no IPv6 BGP routes were being sent to other peers from Quagga's bgpd. I posted a message to the mailing list, and quickly got a helpful response. Apparently I was hitting a bug that's been in Quagga for awhile (typo) that dealt with the address-family negotiation between peers. The quick fix was to add 'override-capability' to each neighbor (or peer group) and it would accept all advertised address families.

After all the peers were setup, I disabled [[OSPFv3]] on all the WAN links, and everything reconverged... oddly. It looked like BGP was doing path-selection based on tiebreakers, and picking the higher peer address as the best path for a destination, even if it meant not utilizing the directly connected link. After scratching my head for a few minutes, I realized my stupidity. Normal BGP treats AS_CONFED_SEQUENCE and AS_CONFED_SET as a length of one, so all paths through my network looked like they had an AS path length of *1*. Luckily, Quagga had a nice bgp bestpath as-path confed command that modified the path selection algorithm, and gave me what I wanted. I described this a blog entry.

Since I wanted all loopbacks and transit interfaces reachable from anywhere, I added a ton of network statements to bgpd. It felt like a hack, but isn't too bad, since there's really no other way of doing it, without using a network-wide IGP.

==== IPv4 Migration ====

Since the IPv6 migration was successful, I figured the IPv4 migration would turn out the same - and it did, mostly.

I started setting up the IPv4 BGP neighbors, and ran into a strange issue with ScreenOS. I've documented it here. Basically, my two Juniper firewalls wouldn't establish IBGP connections unless they were configured as passive neighbors (wait for a connection).

After all the IPv4 BGP connections were up and running, I killed the network-wide IGP process entirely (shut off ospfd/ospf6d on dax and nox), and let everything reconverge. It worked out of the box - success!

I removed the static default routes on my OpenWrt routers, and advertised defaults at each site. No problem there.

==== Finish ====

Although I ran into a number of problems, and probably complicated troubleshooting of my network by an order of magnitude, I think the conversion was worth it. Now if anyone wants to start Xicada 2.0, we can do it right, this time...

=== EBGP Conversion ===

I got sick of confederations, so I just removed the confederation statements and converted all of the inter-site links to straight EBGP.</div></div>

== Applications ==

PCN enables several applications:

* VoIP (via [[SIP]] / G.711u)
* IPv6 Internet access
* Streaming audio

== Lab ==

:''Main Article: [[PCN Lab]]''

The PCN lab is Mark Kamichoff's network proving ground and general hacking arena.

== External Links ==

* [https://www.prolixium.com/mrtgfe PCN MRTG]
* [http://www.prolixium.net/ PCN Home Page]

File:wan.png

2024-01-09T01:17:48Z

Prolixium: Prolixium uploaded a new version of File:wan.png

PCN WAN Architecture

Movies seen by Mark Kamichoff

2024-01-01T23:54:00Z

Prolixium:

Movies seen by Mark Kamichoff

2023-07-23T02:27:47Z

Prolixium:

Movies seen by Mark Kamichoff

2023-07-04T22:14:59Z

Prolixium:

Movies seen by Mark Kamichoff

2023-07-04T22:14:48Z

Prolixium:

[[Mark Kamichoff]] recently started keeping track of what movies he's seen.

== 2007 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0465602/ Shoot 'Em Up] || 2007/09/08 22:00 [[EDT]] || Entertaining || [http://en.wikipedia.org/wiki/Regal_Entertainment_Group Regal Entertainment Group]
|-
|[http://www.imdb.com/title/tt0431197/ The Kingdom] || 2007/09/28 17:35 EDT || Entertaining || [http://en.wikipedia.org/wiki/AMC_Theatres AMC Theatres]
|-
|[http://www.imdb.com/title/tt0465538/ Michael Clayton] || 2007/10/19 20:05 EDT || [[Good]] || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0765429/ American Gangster] || 2007/11/09 19:40 [[EST]] || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0480249/ I Am Legend] || 2007/12/15 21:50 EST || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0465234/ National Treasure: Book of Secrets] || 2007/12/28 20:25 EST || Boring || Regal Entertainment Group
|-
|}

== 2008 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0880578/ Untraceable] || 2008/02/09 20:00 EST || Unsettling || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1060277/ Cloverfield] || 2008/02/14 19:05 EST || Wow || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443274/ Vantage Point] || 2008/02/22 20:15 EST || Predictable || AMC Theatres
|-
|[http://www.imdb.com/title/tt0478087/ 21] || 2008/03/29 19:50 EST || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1091617/ Expelled: No Intelligence Allowed] || 2008/04/23 19:25 EDT || Revealing || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0411061/ 88 Minutes] || 2008/04/25 19:30 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0371746/ Iron Man] || 2008/05/07 20:00 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0367882/ Indiana Jones and the Kingdom of the Crystal Skull] || 2008/05/23 18:45 EDT || Entertaining || AMC Theatres
|-
|[http://www.imdb.com/title/tt0493464/ Wanted] || 2008/06/27 19:55 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0468569/ The Dark Night] || 2008/07/18 21:00 EDT || Awesome (but too long) || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443701/ The X Files: I Want to Believe] || 2008/07/25 19:55 EDT || Blasphemous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0988047/ Traitor] || 2008/09/01 19:50 EDT || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1034331/ Righteous Kill] || 2008/09/12 20:30 EDT || Interesting || AMC Theatres
|-
|[http://www.imdb.com/title/tt0887883/ Burn After Reading] || 2008/09/19 19:20 EDT || Hilarious || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0830515/ Quantum of Solace] || 2008/11/15 15:55 EST || [http://www.imdb.com/title/tt0381061/ Casino Royale] was better || Regal Entertainment Group
|-
|}

== 2009 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0421715/ The Curious Case of Benjamin Button] || 2009/01/09 22:00 EST || Excellent || [http://www.mezcharlotte.com/ MEZ]
|-
|[http://www.imdb.com/title/tt1114740/ Paul Blart: Mall Cop] || 2009/02/06 19:05 EST || Painful, yet humorous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0963178/ The International] || 2009/03/14 16:00 EDT || Banks are evil? || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0473705/ State of Play] || 2009/04/24 19:45 EDT || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0458525/ X-Men Origins: Wolverine] || 2009/04/30 21:20 EDT || Action-packed || [http://www.ayrsleycinemas.com/ Ayrsley Cinemas]
|-
|[http://www.imdb.com/title/tt0796366/ Star Trek] || 2009/05/07 19:45 EDT || [http://www.prolixium.com/mynews?id=839 Good] || Ayrsley Cinemas
|-
|Star Trek || 2009/05/08 16:50 EDT || Good || AMC Theatres
|-
|Star Trek || 2009/05/09 21:30 EDT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt0808151/ Angels & Demons] || 2009/05/15 19:30 EDT || Book was better, except for the end || Regal Entertainment Group
|-
|Star Trek ([[IMAX]]) || 2009/05/20 19:15 EDT || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1049413/ Up] || 2009/06/12 19:00 EDT || Interesting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0417741/ Harry Potter and the Half-Blood Prince] || 2009/08/01 16:30 EDT || Blah || AMC Theatres
|-
|[http://www.imdb.com/title/tt0361748/ Inglorious Basterds] || 2009/08/23 16:00 EDT || Violent || Ayrsley Cinemas
|-
|[http://www.imdb.com/title/tt1136608/ District 9] || 2009/08/29 17:40 PDT || Surprising || [http://www.pacifictheatres.com/ Pacific Theatres]
|-
|[http://www.imdb.com/title/tt1190080/ 2012] || 2009/11/24 12:05 EST || Thrilling || AMC Theatres
|-
|[http://www.imdb.com/title/tt0499549/ Avatar] [[3D]] || 2009/12/26 23:30 EST || Awesome || AMC Theatres
|-
|}

== 2010 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1228705/ Iron Man 2] || 2010/05/07 22:00 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0944835/ Salt] || 2010/07/24 13:50 EDT || Entertaining || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1323594/ Despicable Me] 3D || 2010/07/30 19:10 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1375666/ Inception] || 2010/08/07 15:25 EDT || Intriguing || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1001526/ Megamind] 3D || 2010/11/27 17:50 EST || Quite good || AMC Theatres
|-
|[http://www.imdb.com/title/tt1104001/ Tron: Legacy] 3D || 2010/12/17 16:00 EST || Awesome || AMC Theatres
|-
|[http://www.imdb.com/title/tt0980970/ The Chronicles of Narnia: The Voyage of the Dawn Treader] || 2010/12/23 18:50 EST || Not bad || AMC Theatres
|}

== 2011 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt0970866/ Little Fockers] || 2011/01/02 16:30 EST || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0993842/ Hanna] || 2011/05/07 19:20 EDT || Strange || AMC Theatres
|-
|[http://www.imdb.com/title/tt0458339/ Captain America: The First Avenger] || 2011/08/07 14:35 EDT || Exciting || AMC Theatres
|-
|[http://www.imdb.com/title/tt1509767/ The Three Musketeers] || 2011/11/04 22:25 EDT || Fun || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1568911/ War Horse] || 2011/12/29 15:35 EST || Meh || Regal Entertainment Group
|}

== 2012 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1229238/ Mission: Impossible - Ghost Protocol] (IMAX) || 2012/01/06 22:10 EST || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0477302/ Extremely Loud and Incredibly Close] || 2012/01/20 21:40 EST || Well done || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1591479/ Act of Valor] || 2012/03/02 22:50 EST || Powerful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1232829/ 21 Jump Street] || 2012/03/30 22:00 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0848228/ The Avengers] || 2012/05/05 21:30 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1409024/ Men in Black III] || 2012/06/09 16:10 EDT || Funny || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2215285/ Madea's Witness Protection] || 2012/06/30 16:10 EDT || Humorous || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1345836/ The Dark Night Rises] || 2012/07/27 21:00 EDT || Excellent || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1790886/ The Campaign] || 2012/08/18 16:50 EDT || Funny, but over the line || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1276104/ Looper] || 2012/10/06 19:30 EDT || Strange || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1074638/ Skyfall] || 2012/11/10 17:20 EST || Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0443272/ Lincoln] || 2012/11/22 19:30 EST || Good || AMC Theatres
|}

== 2013 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1707386/ Les Miserables] || 2013/01/01 16:00 EST || Masterpiece || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1790885/ Zero Dark Thirty] || 2013/01/26 16:35 EST || Dramatic || AMC Theatres
|-
|[http://www.imdb.com/title/tt1606378/ A Good Day to Die Hard] || 2013/02/23 17:40 EST || Explosive || AMC Theatres
|-
|[http://www.imdb.com/title/tt1623205/ Oz the Great and Powerful] || 2013/03/23 17:05 EDT || Childish || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1483013/ Oblivion] || 2013/04/19 22:20 EDT || Beautiful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1300854/ Iron Man 3] || 2013/05/10 19:15 EDT || Exciting || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1408101/ Star Trek Into Darkness] (IMAX 3D) || 2013/05/18 12:50 EDT || Enjoyable || Regal Entertainment Group
|-
|Star Trek Into Darkness || 2013/06/15 19:00 EDT || Enjoyable || AMC Theatres
|-
|Star Trek Into Darkness || 2013/06/30 15:20 EDT || Enjoyable || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0770828/ Man of Steel] || 2013/07/04 17:30 EDT || Gratuitous Destruction || AMC Theatres
|-
|[http://www.imdb.com/title/tt1723121/ We're the Millers] || 2013/08/17 20:10 EDT || Funny, but vulgar || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2357129/ Jobs] || 2013/08/19 19:20 EDT || Inspiring || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1411250/ Riddick] || 2013/09/10 19:40 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1837703/ The Fifth Estate] || 2013/10/23 19:30 EDT || Awesome || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1535109/ Captain Phillips] || 2013/10/26 21:15 EDT || Suspenseful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1981115/ Thor: The Dark World] || 2013/11/10 19:00 EST || Mostly Good || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt3063516/ Jackass Presents: Bad Grandpa] || 2013/11/23 19:00 EST || Funny || AMC Theatres
|}

== 2014 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1205537/ Jack Ryan: Shadow Recruit] || 2014/01/25 17:20 EST || Suspenseful || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt1418377/ I, Frankenstein] || 2014/02/15 20:00 EST || Strange, but good || AMC Theatres
|-
|[http://www.imdb.com/title/tt2872732/ Lucy] || 2014/08/02 14:10 PDT || Disappointing || AMC Theatres
|-
|[http://www.imdb.com/title/tt1790864/ The Maze Runner] || 2014/10/11 15:25 PDT || Great || AMC Theatres
|-
|[http://www.imdb.com/title/tt0816692/ Interstellar] || 2014/11/09 14:10 PST || Great || AMC Theatres
|-
|[http://www.imdb.com/title/tt1809398/ Unbroken] || 2014/12/25 19:40 EST || Long, Unsettling || AMC Theatres
|}

== 2015 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt2395427/ Avengers: Age of Ultron] || 2015/05/15 17:30 PDT || Silly || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt0369610/ Jurassic World] || 2015/06/20 13:05 PDT || Greedy || [https://en.wikipedia.org/wiki/Cinemark_Theatres Cinemark Theatres]
|-
|[http://www.imdb.com/title/tt4046784/ Maze Runner: The Scorch Trials] || 2015/09/19 16:15 PDT || Didn't match the book || Cinemark Theatres
|-
|[http://www.imdb.com/title/tt2279339/ Love the Coopers] || 2015/11/25 12:40 EST || Alright || Regal Cinemas
|-
|[http://www.imdb.com/title/tt2488496/ Star Wars: The Force Awakens] || 2015/12/29 13:45 EST || Great || Ayrsley Cinemas
|}

== 2016 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt2277860/ Finding Dory] || 2016/06/25 16:20 PDT || Fun || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt2709768/ The Secret Life of Pets] || 2016/07/09 14:00 PDT || Fun || AMC Theatres
|-
|[http://www.imdb.com/title/tt2660888/ Star Trek Beyond] || 2016/07/31 13:30 PDT || Too Much Action || AMC Theatres
|-
|[http://www.imdb.com/title/tt2387499/ Keeping Up with the Joneses] || 2016/10/29 11:50 PT || Fun || AMC Theatres
|-
|[http://www.imdb.com/title/tt2543164/ Arrival] || 2016/11/19 16:05 PT || Awesome || AMC Theatres
|-
|[http://www.imdb.com/title/tt3183660/ Fantastic Beasts and Where to Find Them] || 2016/12/04 13:30 PT || Alright || AMC Theatres
|}

== 2017 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt3748528/ Rogue One: A Star Wars Story] || 2017/01/02 11:30 PT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt1219827/ Ghost in the Shell] || 2017/04/08 10:20 PT || Good || AMC Theatres
|-
|[http://www.imdb.com/title/tt3896198/ Guardians of the Galaxy Vol. 2] || 2017/06/11 14:15 PT || Excellent || AMC Theatres
|-
|[http://www.imdb.com/title/tt3469046/ Despicable Me 3] || 2017/07/04 14:45 PT || Mildly Funny || AMC Theatres
|-
|[http://www.imdb.com/title/tt2239822/ Valerian and the City of a Thousand Planets] || 2017/07/29 17:10 PT || Decent || AMC Theatres
|-
|[http://www.imdb.com/title/tt1856101/ Blade Runner 2049] || 2017/10/22 1510 PT || Need to rewatch the original || Regal Entertainment Group
|-
|[http://www.imdb.com/title/tt3501632/ Thor: Ragnarok] || 2017/12/02 1540 PT || Funny || AMC Theatres
|-
|[http://www.imdb.com/title/tt2527336/ Star Wars: The Last Jedi] || 2017/12/18 1215 PT || Entertaining || AMC Theatres
|-
|Star Wars: The Last Jedi || 2017/12/25 1930 ET || Entertaining || AMC Theatres
|}

== 2018 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[http://www.imdb.com/title/tt1825683/ Black Panther] || 2018-03-17 1045 PT || Lived up to the hype || AMC Theatres
|-
|[https://www.imdb.com/title/tt4154756/ Avengers: Infinity War] || 2018-04-27 1645 PT || Wow || AMC Theatres
|-
|[https://www.imdb.com/title/tt4123430/ Fantastic Beasts: The Crimes of Grindelwald] || 2018-12-09 1500 PST || Meh || Regal Entertainment Group
|}

== 2019 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt2527338/ Star Wars: Episode IX - The Rise of Skywalker] || 2019-12-25 1515 PST || Decent || AMC Theatres
|}

== 2021 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt1160419/ Dune] || 2021-10-22 2040 PDT || Good || Regal Entertainment Group
|}

== 2022 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt1745960/ Top Gun: Maverick] || 2022-06-25 1615 EDT || Great || [https://en.wikipedia.org/wiki/Alamo_Drafthouse_Cinema Alamo Drafhouse Cinema]
|}

== 2023 ==

{|border="1" cellpadding="2"
!Movie
!Date
!Concise rating
!Theatre franchise
|-
|[https://www.imdb.com/title/tt6791350/ Guardians of the Galaxy Vol. 3] || 2023-05-06 1615 EDT || Good || Alamo Drafhouse Cinema
|-
|[https://www.imdb.com/title/tt1462764/ Indiana Jones and the Dial of Destiny || 2023-07-04 1100 EDT || Good || Alamo Drafhouse Cinema
|}

2022-09-08T03:49:33Z

Prolixium: Prolixium uploaded a new version of File:wan.png

PCN WAN Architecture