█║ S Y N C H R O K N O T ║█

█║▌║▌║ SynchroKnot Instructional Manual ║█

**This document is a work-in-progress and may contain errors. Content can be added/modified anytime.**
*Power Modules have their own separate manuals.
*Familiarize yourselves with ZFS on Linux filesystem and warm up your hands-on skills. SynchroKnot does all the ZFS automation as well, but you will have to create a zpool the first time, expand it when needed, scrub, etc. and replace faulty disks.
*May work with Debian derivatives like Devuan. Some Debian derivatives like Astra Linux and others do not qualify to be used as it is a requirement to have the same Linux Kernel as found in Debian Stretch [i.e Linux kernel version 4.9] and the kernel must accept external non-signed kernel modules [i.e the kernel must not be compiled with module signature verification CONFIG_MODULE_SIG]
*Take your time to understand for a fun learning curve. Good Luck! and ^Zip+++Up^ your Spacesuits!!
  *** You can use any x86_64 hardware with LOCAL DRIVES ONLY. ***
  *** DO NOT import block devices via iSCSI, SAN, NAS, Distributed block/file storage or other methods. ***
  *** DO NOT use Blade Servers. ***
  *** DO NOT run SynchroKnot Software in virtual machines. ***
  *** Use Regular, Inexpensive and High-Performance Commodity Hardware [Eg. workstation/desktop motherboards with AMD [Ryzen] / INTEL [i9]. *** 
  *** Read "GENERIC GUIDE TO HARDWARE SELECTION, CONSULTING AND PRICING" on the website for more information. ***
║█║ Debian 9 [Stretch] on x86_64 Hardware 

    █ Get Started:
    ■  Install Devuan [ASCII] from devuan.org OR Debian 9 [Stretch] from debian.org
    ■  Install Devuan [ASCII] Virtual Machine Image [ Recommended ] OR Debian 9 [Stretch] Openstack Debian Cloud Raw Image on x86_64 hardware
    ╚══> Click Here To Read

        █ Install Standard Packages with Apt. 
        apt-get -y install uml-utilities numactl bridge-utils vlan apache2 qemu-kvm socat ebtables ethtool iptables inotify-tools curl tcpdump dnsmasq lsof ldap-utils nodejs

        *** Only proceed to the next step if the all packages have been successfully installed ***
        ■ If you happen to need a graphical user interface. This is not a requirement but may be helpful.

        apt-get install x-window-system-core gdm3 firefox synaptic
        ■ The known-to-be-working version of bitcoind comes packaged with the SynchroKnot software. Feel free to download and verify the checksum of the latest version from the official Bitcoin site. 
        █ Install and set up ZFS on Linux with a RAID volume of choice with the name tier0 and mounted at /tier0 
        eg. zpool create tier0 /dev/sdb by default mounts at /tier0. 
        ■ If you don't have an extra disk or partition then you can create a file based ZFS volume:
        modprobe zfs
        mkdir /root/zfs
        truncate -s 4000M /root/zfs/zfs0
        zpool create tier0 /root/zfs/zfs0
        zpool scrub tier0
        zpool status

        ■ In some situations, after a reboot you may have to do : 
        zpool import tier0 
        losetup /dev/loop0 /root/zfs/zfs0 && zpool import tier0  
        █ IMPORTANT:

        ■ *** The ZFS tier0 volume *MUST* always be up [online] and mounted at /tier0 before proceeding with any SynchroKnot operation ***
        ■ *** Confirm the workability and testing of the network, for example, checking if your Ethernet cards [drivers and cabling] are set up and working individually and also when connected in ring or torus topology. Verify if vlans [single, double and triple stacks] work when set up [ie 802.1ad+802.1q+802.1q]. This is a requirement every time an operating system is updated/upgraded.***      
        ■ *** Be sure to follow only the QEMU-KVM manual [not libvirt] for syntaxes/options regarding the virtual machine. Wrong options and/or syntaxes may not start/function the virtual machine as expected. When in doubt start a virtual machine using the QEMU-KVM syntax/option[s] from the commandline and then incorporate them into the Spacesuit of the virtual machine using the trigger vm-modify. ***

        If USB Ethernet adapters are used for testing or for whatever reason, then Linksys 3.0 Gigabit Ethernet Adapter [Model No. USB3GIGV1] is known to work [Eg. It does not have to be unplugged and plugged back in while testing bringing down and up ethernet interfaces].
║█║ Spatial Fabric Satellite - 1 Step Deployment: 1] Deploy Synchroknot software : ***IMPORTANT: Always login/logout as a non-root user with SSH [remotely/locally if logged in with GUI/Shell], and then become root and perform the operations below.*** Please ***DO NOT*** login with GUI/Non-GUI directly to perform the operations below or else when you logout the SynchroKnot and other daemons/processes started will be stopped by the operating system. [screen, nohup and other tools & their combination have been tried but it does not seem to resolve this quirk]. a] Install SynchroKnot-Base - This is a one-time operation. Place all the SynchroKnot tar files and SynchroKnot-Base.sknt in a new directory in : /root/synchroknot-latest Then, as the root user do : cd /root/synchroknot-latest && chmod 755 SynchroKnot-Base.sknt && ./SynchroKnot-Base.sknt b] Place the synchroknot.license and other Power Module license[s] [eg. level2-security-power-module.license] in: /tier0/synchroknot/www/synchroknot-modules and do: chown www-data:www-data *.license && chmod 440 *.license ***IMPORTANT: you will see a message depicted below in Log Panorama if the license is not placed appropriately or if a copy of the license is detected across Spatial Fabric Satellites. All SynchroKnot management operations will be halted. The remedy is to have the license in place with the correct permissions as shown above: ..... 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z @ : Illegal License Detected - All Timers Will Be Out Of Sync! : Remove All Illegal Licenses. Example of contents in /tier0/synchroknot/www │ ├── css/ ├── js/ ├── SynchroKnot-Datadecenter-Fabric.sknt ├── SynchroKnot-Identify.sknt ├── synchroknot-modules/ ├── SynchroKnot.sknt ├── tightvnc-jviewer.jar ├── webview.sknt Example of contents in /tier0/synchroknot/www/synchroknot-modules │ ├── disperser-enable.sknt ├── space-debris.sknt ├── spatial-cluster-ctl.sknt ├── synchroknot.license ├── spatial-fabric-satellite-ctl.sknt ├── vm-create.sknt ├── vm-delete.sknt ├── vm-modify.sknt ├── vm-relocate.sknt ├── vm-start.sknt *** Note: The size of largest file is only about 258K! *** *** Simply replace the files to update to a newer version of SynchroKnot without disrupting/stopping any Spatial Cluster[s], Virtual Machines, Remote Consoles, Storage Data Transfers and other aspects! *** c] Set up Spatial Fabric Satellite : echo "trigger:setup ip: ports:eth0,eth1" | /tier0/synchroknot/www/synchroknot-modules/spatial-fabric-satellite-ctl.sknt ***Important: To modify the setup, do a reboot of the Spatial Fabric Satellite and redo the above command with the changes.*** Note: trigger:setup-rt will be released shortly which will allow the non-disruptive modification even if the spatial clusters are active! - Use an IP address only in the 28.xxx.xxx.xxx network. - The SynchroKnot Infrastructure Engine Web Interface access will always be in the 10.xxx.xxx.xxx network corresponding to the IP set in the 28.xxx.xxx.xxx network. Tenant access to this IP address can be given via the internal network or via VPN. Example : If the IP address set above is ip: the SynchroKnot Infrastructure Engine Web Interface access IP will be d] Invoke Spatial Fabric Satellite - Invocation is necessary after the Spatial Fabric Satellite starts [boots] successfully : echo "trigger:invoke" | /tier0/synchroknot/www/synchroknot-modules/spatial-fabric-satellite-ctl.sknt Example output: echo "trigger:invoke" | /tier0/synchroknot/www/synchroknot-modules/spatial-fabric-satellite-ctl.sknt SynchroKnot : Invoking ... Starting Blockchain Gateway : Success Starting Webserver : Success Starting Storage Datapath Access Root : Success Starting SynchroKnot Gateway : Success Starting SynchroKnot Secure Gateway : Success Starting Transmigrate for Spatial Fabric Array : Success Starting Transmigrate for Spatial Fabric Satellite : Success SynchroKnot : Success : Done and Invoked. ║█║ Spatial Fabric Satellite : Post Setup/Modify STEP 1 : Make sure that the ZFS tier0 volume is up [online] and mounted at /tier0 STEP 2 : Make sure that the network cables are in place and then invoke the Spatial Fabric Satellite - Invocation is necessary after the Spatial Fabric Satellite starts [boots] successfully : echo "trigger:invoke" | /tier0/synchroknot/www/synchroknot-modules/spatial-fabric-satellite-ctl.sknt STEP 3 : Proceed as usual with spatial-cluster-ctl.sknt [create, modify, start, force-stop, remove] and other operations as needed. Look at help for more options: echo "trigger:help" | /tier0/synchroknot/www/synchroknot-modules/spatial-fabric-satellite-ctl.sknt *** Available options spatial-fabric-satellite-ctl.sknt are: help|status|invoke|service-restart|enable-network-ports|setup *** trigger:enable-network-ports -- enables and reinstates all the physical network ports [starting with eth and en] that are assigned to the Spatial Fabric Satellite. [In certain rare situations depending on the type of physical machine being used, the network port[s] could go offline when the machine goes into or wakes up from sleep mode, bug in the network driver etc.] ║█║ Spatial Fabric Array : Bifurcated CPU and Memory resources given to a customer / tenant on a particular Spatial Fabric Satellite. The Spatial Fabric Array inherits the IP address [28.xxx.xxx.xxx] of the underlying Spatial Fabric Satellite, and a unique port number is automatically computed from the blockchain ID of the tenant. The same port number is automatically computed for the tenant with the same blockchain ID across all Spatial Fabric Satellites [locally or globally]. This distinct feature allows the internal control and management traffic [unicast/broadcast/multicast] of the tenant to be bifurcated and directed to only the Spatial Fabric Satellite(s) where the tenant exists, and NOT to all Spatial Fabric Satellite(s) as one large unicast/broadcast/multicast domain for all the tenants. *Important: When the key and value spatial-fabric-array:{IP address} is used together with any trigger, then ONLY the Spatial Fabric Array with that IP address hears the trigger. ║█║ Spatial Cluster : Customer / tenant setup, modification, network bifurcation and interconnection of Spatial Fabric Arrays across different Spatial Fabric Satellites. █ Spatial Cluster Create : Create a spatial cluster for the tenant on single or multiple Spatial Fabric Satellites locally or globally dispersed. The tenant can access the SynchroKnot Infrastructure Engine Web Interface on any Spatial Fabric Satellite where the spatial cluster was created and manage any Spatial Fabric Array with the right credentials. The tenant gives the infrastructure provider their Bitcoin ID and a message signed with their private key. The infrastructure provider verifies the signed message. Successful verification proves the tenant is the owner of the private key and the infrastructure provider can go ahead and create the spatial cluster. echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z interstellar:1000 cpu:0,1,2,3 quota:2G trigger:spatial-cluster-create" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt - synchroknot:[Blockchain ID / Bitcoin ID of the customer / tenant] - interstellar:[single OR dual OR triple stacked VLAN. Example : 1000 | 1000.1000 | 1000.1000.1000] - cpu:[comma separated value of the CPU cores you would like the customer / tenant to have] - quota:[size in GB. Example : 500G] *** It is *NOT* necessary to create the same Spatial Cluster on all Spatial Fabric Satellites. *** Eg. if you have 10 Spatial Fabric Satellites, then you can create spatial cluster 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z as needed on all 1 to 10 OR 3 and 8 OR 2,3,6,8,9 etc. All the virtual machines in that spatial cluster will have transparent layer 2 connectivity! *** You have the flexibility to transparently expand and contract the spatial cluster when you need *WITHOUT* disrupting/interrupting the existing Spatial Clusters and the virtual machines already present & running across different Spatial Fabric Satellites. *** █ Spatial Cluster Start : The ZFS tier0 volume must be up [online] and mounted at /tier0 ***Important : As mentioned above, always login in as non-root user with SSH [remotely/locally], and then become root and perform the operations below.*** echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z trigger:spatial-cluster-start" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Example output: echo "synchroknot:1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE trigger:spatial-cluster-start" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt SynchroKnot : Starting Spatial Cluster : 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE SynchroKnot : Success : Spatial Cluster 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE Invoked. █ Spatial Cluster Modify : trigger:spatial-cluster-modify piped to /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt with the options below : add-cpu:[physical cpu has to exist and within the allowed limit mentioned in the license] remove-cpu:[cpu number] tier0-quota:[example: 128G] *Important: spatial cluster has to be modified before being started to reflect the changes. █ Spatial Cluster Force Stop : echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z trigger:spatial-cluster-force-stop" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt A complex attempt will be made to stop all aspects of the specific Spatial Cluster! Note: You will need the key and value am-i-sure:yes for trigger spatial-cluster-force-stop to work. █ Spatial Cluster Blockchain Identification : Find out how many spatial clusters exist on the Spatial Fabric Satellite echo "synchroknot:1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE trigger:scid" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Example output: echo "synchroknot:1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE trigger:scid" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Spatial Cluster Blockchain Identification ----------------------------------------- 18aMrKwYjRc9T6gMoiPgQS4ggSRZqtV3Au 19cTPsdaoYu8w47yC1hqYJrcBWSvF5eZVB 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z 1JsMy66EGn1Yzw9FS5ky2EnhyUHUVuNrDe 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE █ Spatial Cluster Status : Displays Realtime Active/Inactive Status Information [processes/daemons/ports] echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z trigger:spatial-cluster-status" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Example output: echo "synchroknot:1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE trigger:spatial-cluster-status" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Spatial Cluster : 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE Spatial Cluster ID : 43501 Storage Quota : 8G Spatial Cluster Network : ACTIVE Spatial Cluster Listener : ACTIVE on Port : 43501 Storage Data Path : ACTIVE on Port : 21836 Log Panorama : ACTIVE on Port : 27675 :::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: SynchroKnot Secure Gateway : ACTIVE on SynchroKnot Gateway : ACTIVE on SFA Transmigrate : ACTIVE SFS Transmigrate : ACTIVE SFS Storage Data Path : ACTIVE Blockchain Gateway : ACTIVE █ Spatial Cluster Service Restart : echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z trigger:spatial-cluster-service-restart" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt This will only restart SynchroKnot processes/services/daemons related to the specific Spatial Cluster without affecting virtual machines, disperser, power modules and other aspects. The SynchroKnot listening processes/services/daemons of the Spatial Cluster whose service is being restarted may be impaired [usually under a second] from hearing during the restart process. If a push/pull-based data transfer was initiated with Storage Datapath, then it might have to be reinitiated. █ Spatial Cluster Remove : echo "synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z trigger:spatial-cluster-remove" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Note: You will need the key and value am-i-sure:yes for trigger spatial-cluster-force-remove to work. █ Spatial Cluster CPU Map : Displays CPU to Spatial Cluster Mapping Displays the mapping betwwen the CPU cores on the system to the Spatial Cluster. echo "trigger:cpu-map" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt You will notice in the example below that only one or two cpu cores are dedicated to a Spatial Cluster. This was done only for testing purposes to check and push the limits. It should not be done in actual use case scenarios. You should dedicate 4 or more cores per Spatial Cluster depending on the number of virtual machines and their type of workload. Example output: echo "trigger:cpu-map" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Total CPUs : 0 - 3 CPUs : Spatial Cluster ---- : --------------- CPU 0 : 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE CPU 1 : 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE CPU 2 : 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z CPU 3 : - █ Spatial Cluster Interstellar Map : Displays Single/Double/Triple Stacked VLANS to Spatial Cluster Mapping echo "trigger:interstellar-map" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Example output: echo "trigger:interstellar-map" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Spatial Cluster : Single/Double/Triple Stacked Vlans --------------- : ---------------------------------- 18aMrKwYjRc9T6gMoiPgQS4ggSRZqtV3Au : 1000 19cTPsdaoYu8w47yC1hqYJrcBWSvF5eZVB : 1000.1000 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z : 10.100.1000 1JsMy66EGn1Yzw9FS5ky2EnhyUHUVuNrDe : 1000.1000.1000 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE : 9 █ Spatial Cluster Help : Displays Information about License and Triggers echo "trigger:help" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt Example output: echo "trigger:help" | /tier0/synchroknot/www/synchroknot-modules/spatial-cluster-ctl.sknt █║▌║▌║ <<^>> SynchroKnot License ID : 19vy4inWdEfavwtPWb68GPp7WDihEwJ1Fr │ ├──█║▌ Organization ID : 1Gqb8EPWVCmyKqMJKeso6kFRn1ptsmU2LZ │ ├──█║▌ Organization : example-corp │ ├──█║▌ Country : Japan │ ├──█║▌ Authorized CPUs : 64 │ ├──█║▌ Expiration Date : Thu-Jan-16-18:47:25-EST-2020 │ ├──█║▌ Version : 1.0 │ ├──█║▌ Power Module[s] : arpless-interstellar, blockchain-ssh, fastr, interstellar, level2-security, reachout, realtime-cache, synchrosync │ └──█║▌ SynchroKnot : trigger:[help|cpu-map|interstellar-map|scid] ▌ trigger:spatial-cluster-[status | create | modify | start | service-restart | force-stop | remove] ║▌█ ║█║ Microcosm, Macrocosm, Intercosm : Microcosm, Macrocosm and Intercosm can be set and updated by the tenant after the Spatial Fabric Array is created by the infrastructure provider. Microcosm : tag(s) related to where the Spatial Fabric Array is located [eg. row, rack/shelf, CPU type, network type, topology etc]. Macrocosm : tag(s) related to region where the Spatial Fabric Array is located [town, city, state, country, zip code, north, south, east, west, ne, ne, se, sw etc]. Intercosm : tag(s) related to group/team/provider identification [names/blockchain id] for correspondance, management and support, and a combination of Microcosm and Macrocosm. trigger:spatial-fabric-array-modify spatial-fabric-array:{ip} intercosm:{tag(s) up to 15 csv} macrocosm:{tag(s) up to 15 csv} microcosm:{tag(s) up to 15 csv} ║█║ Excerpts : Excerpts are tags given to virtual machines. Excerpts can be set and modified with the trigger:modify with up to 15 comma-separated values having the legal characters: 0-9 a-z , - The most powerful fine-grained search specificity and functionality is invoked with the built-in Parallel Regex Logic & Engine when excerpts are further used in combination with microcosm, macrocosm and intercosm. trigger:excerpts intercosm: macrocosm: microcosm: spatial-fabric-array: synchroknot:{tag(s) up to 15 csv} Clicking on "SYNCHROKNOT" on the Infrastructure Engine or leaving a blank value for the key synchroknot: matches all authorized virtual machines and returns their real-time status and metadata [visible via mouse-over]. Refer to video demonstration for a very basic example. ***Important: If the value for the key[s] intercosm/microcosm/macrocosm are given then excerpts will only return results when their values EXACTLY match. This is true for all matches for intercosm/microcosm/macrocosm for all other triggers other than excerpts as well.*** Excerpts Realtime Cache : trigger:excerpts-rtc : works with Realtime Cache Power Module. Excerpts Regex : trigger:excerpts-regex : allows for powerful regex matches as compared to the exact matches with trigger:excerpts Excerpts Realtime Cache Regex : trigger:excerpts-rtc-regex : works with Realtime Cache Power Module to perform powerful regex matches as compared to the exact matches with trigger:excerpts-rtc Examples of the new regex-based excerpt triggers will be included in this section in the near future. The different SynchroKnot Excerpts triggers utilizing different transports automatically give exceptional control over decentralized virtual infrastructure. ║█║ Distance Vector : The distance vector comes preset with an optimal value of 0.7 [i.e 700 milliseconds/ms]. You can increase or decrease it as required. The Spatial Fabric Array listens and waits for 700 ms before it processes the response[s] to the request[s] sent out. trigger:spatial-fabric-array-modify spatial-fabric-array:{ip required} d-vector:{0.25 to 3.5} Different Spatial Fabric Arrays can have different distance vectors. For example, logging into Spatial Fabric Array with d:vector:0.3 would result in a faster response from local or near-local network(s). Logging into Spatial Fabric Array with d:vector:0.7 [or more] may accomodate a distributed global network. It is not a simple task determining local/global network responsiveness. The response to a trigger from local/global Spatial Fabric Arrays depends on a lot of factors. The distance and the load [the cpu cycles available at the destination to process the trigger quickly] can be considered the main factors. For fun, using the trigger:excerpts after setting NETEM delay of 300ms [Eg. tc qdisc add dev eth0 root netem delay 300ms] with the preset distance vector [700 milliseconds/ms] there is still a proper response, even if a single core is dedicated to a spatial cluster on a 2 core [~2GHz] 2GB RAM hardware at the destination! This is just an example of our stretching the distance with UDP in difficult real-world situations on low performance hardware for the purposes of testing. With the Power Module Reachout and Realtime Cache, the distance can be stretched by 5x to 10x. ║█║ Layer 2 Peer-to-Peer VPN : The infrastructure provider(s) can use any layer 2 VPN [preferably it should be peer-to-peer] to interconnect sites with Spatial Fabric Satellites. ***Satellte Tree Protocol *MUST* be blocked from getting in and out of the VPN. In other words, neither site should let the Satellite Tree Protocol reach the other site[s]. There is no harm if it does, but it might reduce the scalability and increase the optimal path selection time in certain situations, as metadata of the local site will be propagated across all the other sites.*** The tenants can also use layer 2 peer-to-peer VPN inside their virtual machine[s] to simply connect their local/global site(s) separate and independent of the infrastructure provider(s) and not have to do anything related to Satellte Tree Protocol or any other aspect. Tinc is a peer-to-peer VPN which is widely used. It is known to have good performance, plus it is mature and stable. This makes it a good VPN solution to use with SynchroKnot. If you choose Tinc then keep the following in mind : ■ Read : "Example: bridging Ethernet segments using tinc under Linux" [using the switch mode] on the official tinc website. ■ If the total sites connected are 50 - 100 or more, then set up 5 - 15 nodes [or more depending upon your setup] as the "connect to" peers or C2-Peer. Here every Tinc node will only connect to the C2-Peers with the "connect-to=" option, allowing you to scale. Without the C2-Peers, every node might connect to eachother and notify eachother every time a new node joins and leaves, generating a lot of metadata, among other things. This can cause a load on the network, as some nodes might experience spiked cpu, memory and bandwidth usage. ■ Adjust the MTU if needed when allowing single/double/triple stacked vlans to traverse. ■ Use hardware with enough fast cpus and memory, and needless to say, enough bandwidth. ■ Set up a virtual machine for Tinc pre-configured as a Spacesuit for the tenants with instructions on how to quickly interconnect their sites. ■ If different infrastructure providers share sites, then make sure that there is an understanding such that the IP addresses of the Spatial Fabric Satellites [i.e 28.xxx.xxx.xxx range] don't clash. ■ That's it! ║█║ Spatial Fabric Satellite Interconnection Across Multiple Sites : Communication between Spatial Fabric Satellites is untagged [no vlan tags used] with IP addresses assigned in the 28.xxx.xxx.xxx range. Use layer 2 loop-free VPN [see above] to connect dispersed sites/locations. - IP addresses of all Spatial Fabric Satellites must be unique and in the 28.xxx.xxx.xxx range - Layer 2 traffic [mainly, IPv4 UDP unicast, UDP broadcast and TCP from and to the 28.xxx.xxx.xxx range] needs to be transparently reachable across the VPN like the local network. - ***Scenario 2 and 3 seem more optimal from managing, scaling and flexibility points of view without having the infrastructure provider manage tenant network [i.e VPN, co-ordination of vlans across sites, maintenance of tenant inter-site connectivity etc].*** - As mentioned before, It is *NOT* necessary to create the same Spatial Cluster on all Spatial Fabric Satellites. █ Example : Scenario 1 : - *All traffic* except Satellte Tree Protocol is allowed to traverse the VPN. - Each spatial cluster must be created with the same VLAN ID across all sites. - The MTU of the VPN may need to be adjusted to allow single, double or triple stacked vlans to traverse. Plus, since most VPNs may not be able to look into the source/destination fields of the packets/frames with VLAN tags there are chances that those packets/frames might be broadcasted. Please ascertain your use case scenario carefully. 6 dispersed sites with 10 Spatial Fabric Satellites each -- All sites interconnected via a layer 2 VPN by the same infrastructure provider OR separate infrastructure providers with an understanding of the IP address [28.xxx.xxx.xxx] range they can use. Spatial Cluster for tenant ABC is created on all 60 Spatial Fabric Satellites with the *SAME* VLAN ID. Then, tenant ABC can log into any [or all] Spatial Fabric Satellite[s] and manage virtual infrastructure across all 60 Spatial Fabric Satellites across all the sites seamlessly. The virtual infrastructure [ie virtual machines of tenant ABC] across all the 6 sites *WILL* also be connected. █ Example : Scenario 2 : The setup is the same as Scenario 1, except that single, double or triple stacked VLANS and Satellite Tree Protocol are *NOT* allowed to traverse the VPN. Spatial Cluster for tenant ABC is created on all 60 Spatial Fabric Satellites with *DIFFERENT* VLAN IDs. Then, tenant ABC can log into any [or all] Spatial Fabric Satellite[s] and manage virtual infrastructure across all 60 Spatial Fabric Satellites across all the sites seamlessly. The virtual infrastructure [i.e the tenant's spatial cluster] across all the 6 sites *WILL NOT* be connected unless the tenant specifically chooses to. The tenant can run any VPN software [Eg. Tinc] in their virtual machine[s] and gain inter-site connectivity. This gives tenants full interconnection specificity and flexibility. The tenant can connect to any number of sites/locations related and/or not related to the infrastructure provider[s]. The infrastructure provider[s] do not have to manage the tenant network [i.e tenant VPN, VLAN, inter-site connectivity etc]. Read Gateway for Customer / Tenant Inbound and Outbound traffic below for more information. █ Example : Scenario 3 : 2 dispersed sites with 10 Spatial Fabric Satellites each -- The Spatial Fabric Satellites at these 2 sites are *NOT* interconnected [separate infrastructure providers]. Spatial Cluster for tenant ABC is created on all 10 Spatial Fabric Satellites on site 1 and all 10 Spatial Fabric Satellites on site 2. The tenant ABC *CANNOT* seamlessly manage the virtual infrastructure on site 1 and site 2, but can separately manage it by logging in separately [eg. 2 browser tabs] into any [or all] Spatial Fabric Satellite(s) on site 1 and site 2. The tenant can choose to connect virtual infrastructure on site 1, site 2 or any other site with a VPN. ║█║ Gateway for Customer / Tenant Inbound and Outbound Traffic to and from the Spatial Fabric Satellite(s) : One can use SynchroKnot Spaceway [when available], or a standard switch / router, or a Linux box to act as a gateway that adds and removes vlan tags coming in from the Spatial Cluster(s) on the Spatial Fabric Satellite(s). The level 1 tag [i.e base tag] is always 802.1ad [Ethertype 0x88a8] and the level 2 and level 3 tags are 802.1q [Ethertype 0x8100]. *FIRST*, block all traffic going in and coming out of the interface with the vlan tag that is assigned to a customer / tenant, *ONLY THEN* allow the public IP address [IPv4/IPv6] that was alloted to the tenant. ***This means ZERO management of the tenant networks by the infrastructure provider.*** The tenants set up and manage their own VPN. Multiple Ethernet ports from the Spatial Fabric Satellites that are not used as a part of the their interconnect topology can be connected to the gateway(s). Spanning Tree Protocol, Rapid Spanning Tree Protocol or similar other protocol traffic MUST NOT reach the Spatial Fabric Satellites from the gateway(s). The gateway(s) MUST NOT loop traffic back to Spatial Fabric Satellites. Example Scenario : On site A, the infrastructure provider creates 4 spatial clusters with the blockchain id 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z and interstellar 1000.1000.1000 [triple stacked vlan] for the tenant, and then allots public IP address in the range eg. The provider, before giving out this IP range to the tenant, blocks all traffic at the gateway that tags/untags vlan 1000.1000.1000, and only allows the IP range to go in and out of the interface with the vlan tag 1000.1000.1000. That's it! The tenant can then set up their own VPN in a virtual machine and get global direct connection to any number of sites regardless of the underlying provider. Let's say here the client connects to site B and C where the same Spatial Cluster 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z is running on vlans 999.999 and 234 on sites B and C respectively. There is no need to coordinate between the infrastructure providers regarding vlans or any other aspect regarding Spatial Fabric Satellites. Also, there is no need for BGP peering. With the SynchroKnot product, 802.1ad [Ethertype 0x88a8] IS NOT used the same way it is used with Metro Ethernet [i.e service and customer vlan ids] across various connection brokers/providers providing interconnection and BGP peering between various customer and cloud provider sites. With the SynchroKnot product, up to triple-stacked vlans are used in just one site! [How many vlans does that make? Not many?]. In addition to 5G and related technologies, could we have made the provision for pooling all the wired and wireless IoT devices in and around the region of that one site? If you are aware of how this is done today with Metro Ethernet and similar technologies and the amount of complexity, cost and time that is required, then you might feel refreshed! This important aspect of the SynchroKnot product and architecture [though over-shadowed by other advantages], not only allows infrastructure providers to set up tenants quickly, but also allows the tenants to near-instantaneously connect all their sites [local/global with automatic redundant routes] transparent of the underlying infrastructure provider! Does this mean something? The infrastructure provider for example, can charge the tenant(s) for the infrastructure and bandwidth, and now gets the flexible option to choose the least expensive internet bandwidth available, and also load balance inbound and outbound traffic across multiple bandwidth links/connections. ║█║ Infrastructure Engine : The Infrastructure Engine can be accessed by all tenants in the 10.xxx.xxx.xxx range corresponding to the 28.xxx.xxx.xxx range IP address given to the Spatial Fabric Satellite. Eg. If the user is logging in for the first time or has cleared the browser cookies, he/she will be automatically taken to the SynchroKnot-Identify.sknt page to log in and then brought back to the infrastructure engine landing page on success. ***There should be no space between the key and value pair separated by a : [Eg. key:value].*** - To log out use trigger:eject and to get license information use trigger:license. - To clear/undo the trigger use ctrl+z. - To set appropriate screen font size use ctrl- to reduce and ctrl+ to increase the font size. Some browsers come with a default large font size which may not be efficient. - To move the cursor quickly across the input box use ctrl+left/right arrow key. - To move the cursor to the start and end of the input box use Fn+left/right arrow key. *Important : When only using the keyboard, use ctrl + enter key to successfully trigger. This feature greatly reduces the chances of mishaps caused by accidental triggering. Tested on Firefox [52.2.0 and above on Linux 64-bit], should work on modern Webkit-based web browsers with Javascript enabled. The virtual machine console access using Java applet will only work in the browser[s] with version[s] that support the Java applets. To log in use your Bitcoin ID as username, your private key and the Bitcoin ID of the spatial cluster. If you do not exist or your id is disabled you will not be able to log in. *Important : SynchroKnot does NOT connect/use the Bitcoin network or other external network(s) to perform any task. Power Module(s) in the future may use it, in which case it will be explicitly mentioned. Regardless to say, your Bitcoin private key MUST NOT be shared with anyone and you MUST ONLY be logging in to a trusted infrastructure provider. ║█║ Spacesuit : Virtual Machine Spacesuit : This is a template of the virtual machine from which new virtual machines can be created. One can create a spacesuit from an existing spacesuit or a virtual machine, and configure its encrypted configuration vault [also called the spacesuit]. The virtual machine spacesuit is identified by its yellow icon and by its naming scheme. Eg. {name}-ss0, {name}-spacesuit0 etc. Each virtual machine, virtual machine spacesuit [i.e template], and a user [ie. synchroknot with a Bitcoin ID] has an encrypted vault also called spacesuit which stores the configuration and metadata. These configurations can be modified with trigger:vm-modify for the virtual machine and trigger:synchroknot-modify for the user [see below]. ║█║ Spatial Virtual Machine : QEMU virtual machines with KVM enabled, known by their fullname [firstname-lastname], firstname and lastname. The firstname is the name given to it and the lastname is a blockchain ID that is automatically generated along with its private key. For the most part, only the firstname is required when using commands on the infrastructure engine. The fullname is a requirement only in a few cases to prevent undesirous situations and effects, as this is a fully decentralized system with NO central control logic. Make sure you use the correct QEMU and other options where required, and that the virtual machine's operating system supports those options [eg. drivers, crtl-alt-del, power down etc] for successful boot and operations. The MAC address assigned to the virtual machine(s) cannot be spoofed. One may change the MAC address inside the virtual machine but it will not be able to gain network access. This is a built-in security feature. Vlan tags [802.1ad/802.1q] to and from the virtual machines are NOT allowed. This is a built-in security feature. Excessive ARP requests from the virtual machine(s) will be automatically curtailed. This is a built-in ARP flood prevention feature. There are numerous such features built-in to promote end-to-end secure and efficient operations. It is important that you give the virtual machines a unique first name [though multiple virtual machines with the same first name will work, but will cause serious management and operational issues]. Tenants are advised to maintain a list or database of virtual machines and user names that are used. This may be used for auditing against the virtual infrastructure. Eg. {vmname},{spatial fabric array},{nicx-ip},{nicx-gateway-ip},{excerpts},{interstellar and interstellar resonance ids},{location} | {user blockchain id},{privilege},{name},{contact},{spatial fabric array},{location} For real-time monitoring, stats, and alerts Netdata will have to be installed directly in the virtual machines. The tenant will need a path/route [direct/VPN] from his/her machine to access the Netdata port on the virtual machine. Please refer to the Netdata manual. Refer to video demonstration █ Create A new a virtual machine can be created in the following ways: ■ From a spacesuit. This spacesuit may be in a stopped, paused or running state and the new virtual machine created from it will be created without any interruption or disruption to the existing spacesuit. ■ From an existing virtual machine using it as a spacesuit. The existing virtual machine can be in a stopped, paused or running state and the new virtual machine created from it will be created without any interruption or disruption to the existing virtual machine. A spacesuit can be created in the following ways: ■ From an existing spacesuit which may be in a stopped, paused or running state without any interruption or disruption to the existing spacesuit. ■ From a virtual machine which may be in a stopped, paused or running state without any interruption or disruption to the existing virtual machine. The unique features described above bring about operational and management flexibility. A virtual machine is automatically created on a Spatial Fabric Array with the best resources. If a virtual machine needs to be created on a specific Spatial Fabric Array then the key spatial-fabric-array: needs value of an IP address. The spacesuit [template with the yellow icon] or the virtual machine acting as a spacesuit needs to be present there. If not, you can create a virtual machine on a Spatial Fabric Array that has that spacesuit or virtual machine [chosen to be used as a spacesuit], and then [auto/manual] relocate it. For example, you have 5 Spatial Fabric Arrays and all of them have a spacesuit called debian as the firstname and different lastnames, then you can create [auto/manual] virtual machines across all 5 Spatial Fabric Arrays. Now, if you only had spacesuit debian on Spatial Fabric Arrays 1 and 2, then virtual machines can only be created [auto/manual] on Spatial Fabric Arrays 1 and 2, and then relocated [auto/manual] if necessary. ■ Name must be 30 characters or below and can only have these legal characters [numbers, a to z and -] without spaces: - 0-9 a-z ■ The name of the spacesuit from which the virtual machine is to be created can just be its first name if it is an actual virtual machine spacesuit. [It does not matter if the same spacesuit exists on different Spatial Fabric Arrays with different lastnames, SynchroKnot automatically takes care of that complexity.] ■ If a virtual machine is to be created from an existing virtual machine, then the fullname of that existing virtual machine needs to be given as a spacesuit along with origin:spatial-vm ■ If a spacesuit is created from an existing virtual machine, then the fullname of that existing virtual machine needs to be given as a spacesuit along with origin:spatial-vm type:spacesuit █ Delete Virtual machine can be deleted using the delete button from the infrastructure engine. You will be prompted for a confirmation while doing so. If the trigger:vm-delete is used from the keyboard, then the fullname of the virtual machine and the IP address of the Spatial Fabric Array where it is located is needed, and the virtual machine will be removed without prompting for confirmation. As a unique feature, the virtual machine can also be deleted if it is in a running or paused state without having to stop it [or processes attached to it or closing all the open consoles], or unconfigure the SpaceBuoy/DHCPCAST/FASTR/Interstellar/Arpless etc or any other Power Module. All the actual complexity associated with virtual machine deletion including its intertwined interdependencies with snapshots and other virtual machines is automatically taken care of by SynchroKnot. Refer to video demonstration █ Operations start, stop, pause, unpause, unplug, reset -- these triggers work from the keyboard with the virtual machine's fullname, or with the click of a button. Refer to video demonstration. ║█║ Spatial Virtual Machine Replica - Writable Snapshot : A Replica can be created from an existing virtual machine or a spacesuit which may be in a stopped, paused or running state without any interruption or disruption to the existing spacesuit/virtual machine. trigger:create type:replica origin:spatial-vm spacesuit:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 synchroknot:debian-webserver-replica1 *Important : You cannot use the replica if the original virtual machine exists, as it is an identical copy. Fullname of the spacesuit is required. End the virtual machine's replica name with -replica+number Eg. webserver-replica1, webserver-replica8 etc. The virtual machine's replica is identified with a light blue icon. You can create a new virtual machine from the replica [of the original virtual machine] acting as a spacesuit. █ Time Travel with Spatial Virtual Machine : Move back and forth in time with specific granularity to the preferred state of the virtual machine at the point in time when the replica was created. First, delete the original virtual machine or make a replica of the original virtual machine. Then, delete the original machine. After that, create a virtual machine with its original name from the replica acting as its spacesuit! Multiple simple and complex scenarios are possible. ║█║ Disperser Refer to video demonstration. trigger:disperser-create spatial-fabric-array: synchroknot: disperse:, internal-ip: trigger:disperser-enable spatial-fabric-array: synchroknot: trigger:disperser-disable spatial-fabric-array: synchroknot: trigger:disperser-info spatial-fabric-array: synchroknot: trigger:disperser-info spatial-fabric-array: synchroknot: trigger:disperser-remove spatial-fabric-array: synchroknot: █ Inter-Weaving If an IP address of the virtual machine being dispersed to is on a hardware with faster or more cpu cycles, then you could choose to inter-weave that IP address so that it gets more traffic. Eg. disperse:,,,,,, ---- here has faster/more cpu cycles. The disperser will not be able to run on specific cpus given to the tenant [due to its kernel functionality]. ║█║ Synchroknot [User] SynchroKnot is the user identified by his/her Blockchain ID [Bitcoin ID]. Each Synchroknot has a Synchroknot Decentralized Access Control Identification known as SDAC with root and non-root privileges. Synchroknot root user is synchroknot0 [sdac-root] and has all the privileges to control and manage all aspects across all Spatial Fabric Arrays. When a spatial cluster is created on the Spatial Fabric Satellite, the first user with the blockchain id of the spatial cluster is automatically created. The tenant can log in to the Spatial Fabric Satellite with the spatial cluster's blockchain id as the user id and the private key, and then create new users. ║█║ Synchroknot Decentralized Access Control Identification - SDAC and VM-SDAC Automatic decentralized real-time ad-hoc calibration of credentials of the Synchroknots [SDAC] with those of the virtual machines [VM-SDAC] for fine-grained decentralized access, control and management of the virtual infrastructure. SDAC ids and VM-SDAC ids start with the name synchroknot and end with numbers [eg. synchroknot101]. A Synchroknot with SDAC-Root has the privilege to access, manage and control all the virtual machines and the spacesuits of the other Synchroknots. For the non-root Synchroknots to access, manage and control all the virtual machines, their SDAC-ID must pass successful calibration with the VM-SDAC id of the virtual machine. Each synchroknot and virtual machine can have up to 15 [csv] SDAC and VM-SDAC ids respectively. For example, if any one of the 15 SDAC ids of the Synchroknot calibrates successfully with any one of the 15 VM-SDAC ids of the virtual machine, then that Synchroknot can access, manage and control that virtual machine, and also have the ability to create new virtual machines and replicas from it. Examples: Synchroknot with sdac-id:synchroknot112 will be able to access, manage and control all virtual machines that have vm-sdac-id:synchroknot112. OR vm-sdac-id:synchroknot25,synchroknot112,synchroknot19 and so on. Synchroknot with multiple sdac-ids Eg. sdac-id:synchroknot112,synchroknot25,synchroknot19 will be able to manage and control all virtual machines that have vm-sdac-id:synchroknot112,synchroknot101,synchroknot25,synchroknot19 all together OR separately vm-sdac-id:synchroknot112 | vm-sdac-id:synchroknot25,synchroknot22222 | vm-sdac-id:synchroknot4555,19 OR in any other combination!! Only SynchroKnot-Root user can modify the SDAC and VM-SDAC ids. █ synchroknot-create trigger:synchroknot-create spatial-fabric-array:[SFA IP] synchroknot:[Bitcoin ID] trigger:synchroknot-create spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z ***You need to have the sdac-root:synchroknot0 privilege to use the trigger:synchroknot-create. The user created must possess the private key associated with the Bitcoin ID, if not he/she will not be able to log in.*** *value for the key spatial-fabric-array: is needed* █ synchroknot-info trigger:synchroknot-info trigger:synchroknot-info spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z *value for the key spatial-fabric-array: is needed* *Full details will only be viewable by the synchroknot-user to whom the Spacesuit belongs and to the users who have the sdac-root:synchroknot0 privilege. All other synchroknot users will see minimal details.* █ synchroknot-list trigger:synchroknot-list trigger:synchroknot-list spatial-fabric-array: *value for the key spatial-fabric-array: is optional depending on your search criteria* █ synchroknot-remove trigger:synchroknot-remove spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z ***You need to have the sdac-root:synchroknot0 privilege to use the trigger:synchroknot-remove. User with sdac-root:synchroknot0 privilege cannot remove himself, but can remove others as needed.*** *value for the key spatial-fabric-array: is needed* █ synchroknot-modify trigger:synchroknot-modify spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z sdac-id:synchroknot19 ***You need to have the sdac-root:synchroknot0 privilege to: -remove or add the sdac-root:synchroknot0 privilege -assign/modify multiple sdac-ids -enable/disable status [ie. status:enabled]. *value for the key spatial-fabric-array: is needed* ***To assign a non-root privilege to a user, *Make Sure* the key sdac-root: has no value given to it.*** ***Those without sdac-root:synchroknot0 privilege *WILL NOT* be able to modify themselves to become sdac-root OR modify their sdac-id.*** The synchroknot-modify trigger below enables user 1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z with sdac-id:synchroknot112,synchroknot25,synchroknot19, while removing synchroknot0 privilege by leaving the value of key sdac-root blank. The trigger below will only be successful if it is initiated by someone with sdac-root:synchroknot0 privilege. trigger:synchroknot-modify spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z sdac-id:synchroknot112,synchroknot25,synchroknot19 sdac-root: status:enabled To disable a synchroknot you must be synchroknot0 : trigger:synchroknot-modify spatial-fabric-array: synchroknot:1J3XTYzPeHGPpY7pezBioBSXnahtDLjS7z status:disabled ║█║ Spatial Fabric Array Information and Log Panorama █ spatial-fabric-array-info trigger:spatial-fabric-array-info spatial-fabric-array: synchroknot: *value for the key spatial-fabric-array: is needed* Returns information about the Spatial Fabric Array. Click on the displayed IP:Port of Log Panorama to open it in a new tab. Use the username and password provided in the information when logging in the first time. █ spatial-fabric-array-list Shows a list of Spatial Fabric Arrays along with their information. It can be used in combination with microcosm, macrocosm and intercosm. Clicking on the returned Spatial Fabric Array information will open Log Panorama in a new tab. When logging in for the first time you will be prompted for the username and password which is available in the URL bar of the web browser! ║█║ Stagnant Virtual Machine A virtual machine becomes stagnant when its metadata is disabled. The stagnant virtual machine will not show up when using trigger:excerpts or other triggers. The metadata of the virtual machine can be disabled when for example its relocation fails, in which case you are likely to see two virtual machines with the same fullname on two separate Spatial Fabric Arrays. The destination virtual machine, which was waiting for the source to relocate, can be made stagnant as a precautionary measure so that it does not, by some chance, respond to triggers, and then it may be safely deleted. trigger:stagnant-vm spatial-fabric-array: synchroknot: trigger:enable-metadata spatial-fabric-array:[SFA IP] synchroknot:[vm firstname or firstname-lastname] trigger:enable-metadata spatial-fabric-array: synchroknot:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 trigger:disable-metadata spatial-fabric-array: synchroknot:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 ║█║ Zombie Virtual Machine If a virtual machine has ■█>Z-0-M-B-I-E<█■ shown before the virtual machine when the trigger:stagnant-vm is used then use the trigger below to remove it: trigger:zombie-vm-delete spatial-fabric-array: synchroknot:testvm-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dknN1 *value for the key spatial-fabric-array: is needed* ***Make sure you are removing a zombie virtual machine with trigger:zombie-vm-delete and not any other virtual machine.*** SynchroKnot finds zombie virtual machines on deeper inspection of the ZFS volume. In certain cases they may have lingered around after their deletion due to a variety of technical reasons. ║█║ Relocate Virtual Machine ■ Manual Relocation Refer to video demonstration trigger:relocate destination-spatial-fabric-array: synchroknot:test123-13ttthWjYmTMKtHxx5f7DJ23PpMDri1PHi ■ Auto Relocation Refer to video demonstration trigger:relocate synchroknot:test123-13ttthWjYmTMKtHxx5f7DJ23PpMDri1PHi To auto select an optimal Spatial Fabric Array with high performance use key and value : performance:high To auto select an optimal Spatial Fabric Array with low performance use key and value : performance:low *Important: Fullvmname [i.e firstname-lastname] is needed when using the keyboard to trigger:relocate. For fully automatic relocation hit the "AUTO-RELOCATE" button on the Infrastructure Engine. *Important: Only the disks attached to the virtual machines will be relocated with the virtual machine. This unique feature prevents build-up of non-relevant storage which costs the industry exabytes of wasted storage, as not everything is de-duplicated mainly for the reasons of performance and portability. After successful relocation the virtual machine on the source Spatial Fabric Array WILL NOT exist [as it will exist on the destination, so don't be surprised to see a trigger vm-delete successful after trigger vm-relocate successful in realtime log panorama]. To keep all the storage after successful relocation on the source for whatever reason, use the key and value data:preserve with the trigger:relocate. SynchroKnot automatically makes the source virtual machine stagnant when data:preserve is used. Eg : trigger:relocate data:preserve destination-spatial-fabric-array: synchroknot:test123-13ttthWjYmTMKtHxx5f7DJ23PpMDri1PHi *Important: To prevent failure in virtual machine relocation please make sure the following conditions are adhered to: a] Before relocating, the virtual machine's cdrom value must be made blank. If the cdrom is attached, then it must be detached. b] Any open console[s] [java applet, html5, vnc/spice client etc] used to access the virtual machine must be closed. c] The destination must not have a stagnant copy of the virtual machine. d] Please do not perform any operations [pause, unpause etc] while it is relocating. e] Please do not symlink the virtual machine's disk[s] outside the ZFS volume where it was automatically placed at the time of creation. Virtual machines with symlinked disk[s] will not relocate. *Important: Virtual machines with interstellar and interstellar resonance ids enabled *Must Only* be relocated to Spatial Fabric Arrays that have the Interstellar/ARPless Interstellar Power Module present and active on the Spatial Fabric Satellite. ║█║║║ Modify Virtual Machine These are the legal characters when using modify and for most triggers without the brackets: [+ ; = % , . _ 0-9 a-z A-Z -] When you need to insert a space use %20 or # on the Infrastructure Engine. Eg. information:This%20is%20SynchroKnot OR information:This#is#SynchroKnot SynchroKnot tries to keep end-to-end Restfulness, but it is difficult as the same Restfulness clashes with the arguments and variables of QEMU, ZFS and other tools. As a built-in security feature aimed at protecting against injection attacks and keeping spatial clusters of tenants securely bifurcated, certain QEMU options may not work or are not supported. Please refer to the QEMU manual for additional details wherever you see [QEMU option(s)]. This will allow one to be tuned with the changes, work-arounds, and new features, etc. directly from the source upstream. █ ***IMPORTANT: Using certian QEMU options require specific drivers to be present in the virtual machine before you can use those options. It is recommended that those drivers are installed in the virtual machine[s] beforehand, and tested on the QEMU commandline to ensure that they work as expected before you modify the virtual machine Spacesuit.*** Multi-order edits are possible with the help of the built-in Intelligent Modification Logic and Engine. You should be able to edit in any order or combination all at once, and all the respective configuration(s) and service(s) will automatically be updated by SynchroKnot! ■ vm-sdac-id vm-sdac-id:[csv synchroknot[0-9]*] -- and up to 15 comma separated values [csv]. Please keep synchroknot[numbers] sane and not too big. vm-sdac-id:synchroknot100 vm-sdac-id:synchroknot100,synchroknot4000,synchroknot33,Synchroknot2000 ■ Enable Replica replica:enable ■ Disable Replica replica:disable ■ Memory memory:2G ■ CPU cpu:[QEMU option(s)] ■ USB Device usbdevice:[QEMU option(s)] ■ Keyboard keyboard:[QEMU option(s)] ■ SMP smp:[QEMU option(s)] ■ Location eg. location:boston location:Washington%20DC location:Washington#DC Legal characters: % - _ 0-9 a-z A-Z ■ Excerpts Up to 15 comma separated values. Legal characters: 0-9 a-z , - eg. excerpts:linux,webserver,production ■ Spice Console Password spice-password:[less than 15 characters.] ■ VNC Console Password vnc-password:[less than 25 characters.] ■ Boot Order boot:[QEMU option(s)] Eg. QEMU option -boot order [ -boot order=ndc first tries to boot from network, then from the first CD-ROM drive, and finally from the first hard disk. a, b (floppy 1 and 2), c (first hard disk), d (first CD-ROM), n-p (Etherboot from network adapter 1-4) ] ■ CDROM The iso must already be present in the virtual machine's storage volume to be attached. cdrom:[name of the iso] -- will attach the cdrom when the virtual machine is started from the stopped state. cdrom: -- blank value removes the name of the iso. cdrom-attach:[name of the iso] -- will attach the cdrom to a running virtual machine. The cdrom:[name of the iso] must be set before that. cdrom-detach: -- detaches the attached cdrom from a running virtual machine. *** Important: Before relocating, the virtual machine's cdrom value must be made blank. If the cdrom is attached then it must be detached. *** ■ Machine machine:[QEMU option(s)] ■ Soundhw soundhw:[QEMU option(s)] ■ VGA vga:[QEMU option(s) eg. std|cirrus|none] ■ Remove Migration Tag In some failed virtual machine relocations the virtual machine might show that it is still migrating, but it might not have migrated due to an underlying failure. So, to remove this tag do: migration-tag:remove ║█║║║ Network Operations ■ Add NIC add-nic:[single or csv from 0 to 49] Eg. add-nic:1 OR add-nic:1,2,3,4,5,6,7,8,9 Note: change the NIC status to plugged after adding it for it to show up in the virtual machine. Read more about NIC Status below. ■ Delete NIC Eg. delete-nic:1 OR delete-nic:1,2,3,4,5,6,7,8,9 ■ NIC Status - Plugged and Unplugged Once the NIC has been added with add-nic it will not be plugged into the virtual machine unless one plugs it in. The results of plugging and unplugging NIC(s) take effect at the starting of the virtual machine from the stopped state. Operating systems may benefit from recognizing the NIC that was plugged back after being previously unplugged, and might keep the same ordering, name and other details, reducing the chances of misconfiguration. nic1-status:plugged nic1-status:unplugged add-nic:1 nic1-status:plugged It is also possbile to add an NIC, plug it in and assign it an IP address, Netmask and all the SpaceBuoy options [see below] all together in one shot! add-nic:1 nic1-status:plugged nic1-ip: nic1-gateway-ip: ■ NIC Model nic[0-49]-model:[QEMU option(s)] Eg. nic0-model:e1000 ■ NIC MAC Address nic[0-49]-mac:[Last 5 hexadecimal characters of a mac address] Eg. nic0-mac:12345 ■ NIC Interstellar MAC Address nic[0-49]-interstellar-mac:[First 7 hexadecimal characters of the MAC address with dac as a prefix] Eg. nic0-interstellar-mac:dac0001 ■ NIC Interstellar Resonance nic[0-49]-interstellar-resonance:[First 7 hexadecimal characters of the mac address with dac as a prefix and up to 10 resonance ids csv] nic0-interstellar-resonance:dac0002 nic0-interstellar-resonance:dac0002,dac0004 *** Interstellar Resonance ids can be anything other than nic[0-49]-interstellar-mac. *** ■ INTERSTELLAR-OUI - This works with the Interstellar/Arpless Interstellar Power Module interstellar-oui:[6 hexadecimal characters of the mac address of the dhcp server i.e OUI of the physical or virtual machine where the dhcp server runs] ║█║ SpaceBuoy ■ NIC IP Address - add public or private IP address to NIC using SpaceBuoy automatically. nic0-ip:[ip address in the 10|172|192 range] -- Eg. nic[0-49]-ip Eg : trigger:modify synchroknot:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 nic0-ip: nic0-gateway-ip: Note: Assigning a public IP address may work for you. Assigning public IP addresses starting with 101 to 110 are known to not work [i.e the virtual machine does not receive these IP addresses]. There may be other public addresses that may not work, but for the most part it seems to be working fine. Please take a look at one of the demo videos to get a better understanding. *Important: 1] To reinstate all the additional options below into the SpaceBuoy, it is required to add or re-add the NIC IP Address [eg. nic[0-49]-ip:] 2] The operating system [eg. DHCP client and related aspects] must support the standard additional SpaceBuoy features listed below for them to get enabled. ■ NIC IP Gateway Address - add public or private IP gateway address using SpaceBuoy automatically. nic0-gateway-ip:[gateway ip address in the 10|172|192 range] -- Eg. add-nic[0-49]-gateway-ip ■ NIC Netmask - add netmask for the NIC IP Address. nic0-netmask:[valid legal netmask] -- Eg. nic0-netmask: ■ NIC Broadcast - add broadcast for the NIC IP Address. nic0-broadcast:[valid legal broadcast] -- Eg. ■ NIC DNS Server -- add up to five DNS Server IP addresses separated by a comma nic0-dns-server: ■ NIC Domain Name - add a domain name nic0-domain-name:[domain name] -- Eg. nic0-domain-name:synchroknot.everywhere ■ NIC Domain Search nic0-domain-search:[domain name] ■ NIC NTP Server nic0-ntp-server: ■ NIC MTU - add a MTU nic0-mtu:[legal MTU size] ■ NIC Classless Static Route - Set a classless static route for the network interface. nic0-classless-static-route: ■ NIC IP Forward Enable - Enable IP forwarding for the interface. nic0-ip-forward-enable: ■ NIC Log Server nic0-log-server: ■ NIC Netbios NS nic0-netbios-ns: ■ NIC Netbios DD nic0-netbios-dd: ■ NIC Netbios Node nic0-netbios-nodetype: ■ NIC SMTP Server nic0-smtp-server: ■ NIC Pop3 Server nic0-pop3-server: ■ NIC TCP Keepalive nic0-tcp-keepalive: ║█║ DHCPCAST ■ To enable DHCPCAST for a network interface: nic[0-49]-dhcpcast:[mac address of the destination dhcp server without the : separator] Eg. trigger:vm-modify synchroknot:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 nic0-dhcpcast:dac000143797 ■ To disable dhcpcast, simply set a blank nic[0-49]-dhcpcast: OR nic[0-49]-ip: Eg. trigger:vm-modify synchroknot:debian-webserver-1NXddcqaWxTyRLJa8S18kFTMP3Zk3dkMm9 nic0-dhcpcast: *Important: If you use the Interstellar Power Module, the destination dhcp server should have the same Interstellar ID or Interstellar Resonance ID. In the case where you happen to use the Interstellar Power Module and the dhcp server cannot have the same Interstellar ID or Interstellar Resonance ID, then one can set the interstellar-oui. ║█║║║ Storage Operations ■ Disk Attach Slot disk[0-24]-attach-slot:[name of the virtual disk. This virtual disk must be present in the virtual machine's storage volume. The attached disk will be seen by the operating system when it is started from a stopped state. It would be logical to keep the disk name as same as the disk[0-24]-attach-slot name if possible.] Eg. disk1-attach-slot:disk1 █ IMPORTANT: *** When a disk is attached it is automatically given the QEMU option of type as virtio, format as raw and cache as writeback. These options can be changed. The raw format [with type virtio] is known for its performance and works well on ZFS. QEMU is known to not start after removing the writeback option. It is recommended that these options are not tweaked. *** *** Please make sure the virtio drivers are pre-installed in the virtual machine if you would like it to boot with the type virtio. *** Below are the options to modify the type and cache: disk[0-24]-type:[virtio|ide|scsi refer QEMU option(s)] Eg. disk1-type:raw disk[0-24]-cache:[writeback refer QEMU option(s)] Eg. disk1-cache:writeback ■ Disk Detach Slot disk[0-24]-detach:[name of the virtual disk. This virtual disk must be present in the virtual machine's storage volume. The detached disk will not be seen by the operating system when it is started from a stopped state.] Eg. disk1-detach-slot:disk1 ■ Disk Format disk0-format:[raw or qcow2 refer QEMU option(s)] ---- disk[0-24]-format -- up to 25 virtual disks supported. Eg. disk0-format:raw ■ Create Virtual Disk trigger:synchrosync-create power-module:synchrosync spatial-fabric-array:{SFA IP} size:{[0-9]K,M,G,T,P,E,Z,Y} synchroknot:[spatial-vm/{vm fullname}/storage/{disk name} | resource-volume/{disk name}] ■ Remove Virtual Disk trigger:synchrosync-remove power-module:synchrosync spatial-fabric-array:{SFA IP} synchroknot:[spatial-vm/{vm fullname}/storage/{disk name} | resource-volume/{disk name}] ║█║ Storage Datapath Access Storage Datapath Access allows you to push and pull data [virtual machine disks, ISOs, spacesuits] to and from any Spatial Fabric Array. This can also be used as a pull-based back-up and/or archiving system. Another great use of Storage Datapath Access is the ease with which one can perform secure, controlled updating and versioning of spacesuits. For example, you can keep the main/original copy of the virtual machine spacesuit(s) at your location [head office, lab, control center, etc.], update the image with security, package and operating system updates/upgrades, test the functionality end-to-end, and then push it [only changed blocks] to as many Spatial Fabric Arrays that have those spacesuit(s)!! █ To access data, simply use : trigger:spatial-fabric-array-info and look for S-PATH:[number] which is the port number of the Spatial Fabric Satellite trigger:spatial-fabric-array-info spatial-fabric-array:[IP address] and look for port next to Storage Data Path trigger:datapath-secrets-info to get the username and password then do: rsync -a rsync://sknt0@ --password-file=[path to your file that contains the password that corresponds to the username.] spatial-fabric-array-log SYNCHROKNOT SPATIAL FABRIC ARRAY LOG ACCESS: Logs for 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE @ tier0-spacesuit SYNCHROKNOT TIER0 STORAGE DATAPATH ACCESS: virtual machine spacesuit for 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE @ tier0-spatial-vm SYNCHROKNOT TIER0 STORAGE DATAPATH ACCESS: virtual machines for 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE @ tier0-resource-volume SYNCHROKNOT TIER0 STORAGE DATAPATH ACCESS: resource volume for 1PeDaGo6Uj9HWdVQs8zJAYeQ1478xrpwLE @ Note: You can then access all the volumes displayed and push/pull data. Above is just an example output to give you an idea of what to expect. ***Important: Make sure only the user has reading rights to the file for it to work, eg. chmod 400 root [yourpasswordfile]*** █ To set/modify the password of datapath secrets use the trigger:datapath-secrets-modify. Up to four users [usernames and passwords] are allowed for multiple access to Storage Datapath. Eg. trigger:datapath-secrets-modify sknt0:yourpassword Eg. trigger:datapath-secrets-modify sknt0:yourpassword sknt1:sknt1password sknt2:sknt2password sknt3:sknt3password ║█║║║ Spatial Fabric Satellite : Deployment Options Cubicles, offices, shops, apartments, basements, wireless base stations, fibre optic hubs/stations etc. instead/alongside centralized data centers. Keep Spatial Fabric Satellites with virtual machines that communicate with each other with frequent/high network I/O in proximity to each-other [Eg. within the same location i.e shelf/rack or close enough location[s]]. The network speed, capacity and bandwidth requirements mainly depend on the type of applications running in the virtual machines and the rate at which the infrastructure is scaled horizontally and/or vertically. ║█║║║ Built-in Switch with SATELLITE TREE PROTOCOL : Satellite Tree Protocol is now available for free on the SynchroKnot Official Website along with an article on how to learn and use it! SynchroKnot software comes built-in with an *ACTUAL* switch with Satellite Tree Protocol. It allows Spatial Fabric Satellites [i.e any commodity X86_64 hardware] to be directly connected to eachother utilizing proven network topologies and without having to configure anything! SynchroKnot software enables and transforms commodity hardware into an *ACTUAL* switch [eg. as seen in products from Cisco, Juniper and others], and not virtual switches like Open Virtual Switch [OVS] and related software. With this unique built-in automatic functionality, if you prefer, you *DO NOT* need physical or virtual switches anymore. Don't forget to see and learn from the demonstration videos and the explanations under the demo section of the website. To disable it for whatever reason, simply turn off STP with the brctl utility on all Spatial Fabric Satellites that see eachother on layer 2 : brctl stp synchroknot0 off Note: after disabling Satellite Tree Protocol you can use standard Ethernet switches and enable STP/RSTP etc on them as needed to connect Spatial Fabric Satellites. Channel Bonding is not supported on Spatial Fabric Satellites. █ The source code of the bridge kernel module should as usual be available at kernel.org or other locations. Check the kernel version on Debian Stretch and then look for the source code of that kernel. Engineers at Spatial Systems Engineering are in the initial phases of looking into gathering requirements for the possibilities of opensourcing the enhancements/modifications made to the kernel module. Feel free to not use it for whatever reason that deems fit. █ If you would like to use and benefit from the built-in Satellite Tree Protocol then: ■ Make sure your Ethernet ports [NIC/USB] are functional. ■ ***Important : Make sure Spanning Tree Protocol / Rapid Spanning Tree Protocol *DO NOT* reach Spatial Fabric Satellites.*** ■ ***You should not need more than 4 Ethernet ports per Spatial Fabric Satellite unless you intend to use custom topologies. Only 2 Ethernet ports are needed for Ring topology and 4 Ethernet ports for Torus topology. Apart from Ethernet ports utilized in the direct connection of Spatial Fabric Satellites, an extra Ethernet port will be needed on those Spatial Fabric Satellite[s] that will be used to connect to your gateway[s]*** ■ To reduce uncertainties in the network, please do not connect Gigabit Ethernet port[s] to a 10 Gigabit Ethernet port[s]. ■ Choose the topology that suits your requirement below. An advanced topology guide may be available on request!
2 Port NIC for 1-D Ring Topology [X Axis] and 4 port NIC for 2-D Torus Topology [X and Y Axis].

Seamless Incremental Network Expansion of 1-D Ring Topology [using both one long cable and single length cable].

2-D Torus Topology.

Seamless Incremental Network Expansion of X and Y Axis.

Multipath Optimized Routes and Links for Mission Critical Operation.

║█║║║ SynchroKnot Auto NAT Enablement SynchroKnot Auto NAT Enablement allows for transparent access to Infrastructure Engine and Virtual Machine Consoles [HTML5/Java], Log Panorama and more from behind NAT [Network Address Translation]. This feature allows for secure and easy setup & access from behind standard NATs so that tenants can have direct access, or access from their VPNs without accessing the actual provider network. This feature brings about flexibility and simplicity, while at the same time allows the service providers to securely keep the tenants separated. Excerpt from the SynchroKnot Manual: The Infrastructure Engine can be accessed by all tenants in the 10.xxx.xxx.xxx range corresponding to the 28.xxx.xxx.xxx range IP address given to the Spatial Fabric Satellite. Eg. To access the SynchroKnot Infrastructure Engine on a Spatial Fabric Satellite from the above description, the IP address of the machine used to access the Infrastructure Engine from the web browser must be in the 10.x.x.x range for security reasons. If you have a tenant behind a transparent NAT in the 172.x.x.x range for example, and it is pointed to the 10.x.x.x range to access the Infrastructure Engine, then the access is possible but with certain limitations. The Infrastructure Engine will not know about the 172.x.x.x range from where the request is coming in as it is on the other side of NAT. Therefore, the response[s] given would still be pointing to the 10.x.x.x range. This would cause the http and other requests & redirects such as Cross Domain Ajax, the opening of new tabs for websocket based HTML5 console access, Java based console access, Log Panorama .... and much more to NOT work. Example scenarios without the use of SynchroKnot Auto NAT Enablement: ■ Scenario A Works: [ web browser in 10.x.x.x range ] ------------> [ Infrastructure Engine in 10.x.x.x range ] Different types of redirects sent from the Infrastructure Engine work transparently. ■ Scenario B Does Not Work: [ web browser in 172.x.x.x range ] -----> [ NAT ] ------> [ Infrastructure Engine in 10.x.x.x range ] [ web browser in 192.x.x.x range ] -----> [ NAT ] ------> [ NAT ] -----> [ Infrastructure Engine in 10.x.x.x range ] Different types of redirects sent from the Infrastructure Engine are pointing in the 10.x.x.x. The web browsers in 172 & 192 ranges behind single and double NATs will trigger the redirect in the 10.x.x.x range but will not be able reach the destination in the 10.x.x.x range. The SynchroKnot Auto NAT Enablement feature transparently addresses this issue and allows full access just as if you were accessing the Infrastructure Engine from the 10.x.x.x network. The above Scenario A and B would work with SynchroKnot Auto NAT Enablement. This should work with any transparent NAT [eg. with IPtables etc]. This unique solution was possible with the combination of partly server-side + partly server-side-embedded-client-side functionality [which is unique to SynchroKnot]. One does not have to touch the transparent firewall! Eg. If you were using IPtables to DNAT and SNAT/Masquerade on a transparent NAT box inbetween, then simply set the 172.x.x.x range to point to 10.x.x.x. range. That's it. No need for further reconfiguration, updating the rules for mapping/unmapping/remapping ports, IP addresses etc. [We will share the details of how to set up a transparent IPtables NAT in our manual shortly.]

Creative Commons License
SynchroKnot [ website - content - technology - architecture - methodology ] by SynchroKnot is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Based on a work by its creator Mehul Sharma at SynchroKnot : synchroknot.[com|org|tokyo|in|ru|ch|de].