Commit Graph
303 Commits
Author SHA1 Message Date
JamesBhattarai b7dbcaf315 Fix: remove hardcoded vnc vm port & remove consts 2026-05-25 19:07:25 +05:45
JamesBhattarai 0e59fee995 Fix(Chore): gpu infra, passthrough 2026-05-25 19:00:47 +05:45
JamesBhattarai 71379950bc Chore: Remove unnecessary constants 2026-05-25 19:00:47 +05:45
coryHawkvelt a71c74f133 Merge pull request 'Implement Openstack Style Metadata with Network ID population' (#49) from JamesBhattarai/theapi:v1.01/network into master
Reviewed-on: coryHawkvelt/theapi#49
2026-05-25 03:35:37 +00:00
JamesBhattarai f1d47cb3a2 Feat(Fix): Implemented Mac, Network ID for MDS lookup 2026-05-24 19:38:40 +05:45
JamesBhattarai e60271d1b9 Fix: make pod_id nullable
Essential for vm creation (as no seperate table for vm)
Exception handle for pod_id enforcer
2026-05-24 17:29:41 +05:45
JamesBhattarai ec106a992e Chore/Fix(network): MDS and Network Dispatch
autostart mds in nscontroller
added missing docker volume/packages for socket mounts for ns
added network id env for nscontroller
add missing module of sdn_helper
error handle when ovs-ofctl is empty
2026-05-24 17:22:01 +05:45
JamesBhattarai cc5f91728a Chore(images): enable format for image's user input 2026-05-24 17:16:10 +05:45
JamesBhattarai 38ddcb5d4d Chore: appropos removed constants
Removed redundant code level constatnt with repo based runtime_urls.py
Added run-master, run-worker.sh scripts for convinient clean, builds
2026-05-23 19:20:18 +05:45
JamesBhattarai 7b685d3580 Chore(Fix): Throw Error when checksum not supplied for volume with image remote/http* 2026-05-23 11:14:50 +05:45
JamesBhattarai bf47406b59 Chore(Fix): added volume http image handler and Removed NorthSouth IP Filter 2026-05-23 02:15:53 +05:45
JamesBhattarai 18b2ec006c Fix(Chore): Infra utils
missing package, docker race state
throw errors in metadata proxy failure in worker
merge all the migrations
add missing field in VmMetadata
2026-05-22 09:34:07 +05:45
JamesBhattarai 80975fb953 Feat: Implemented Metadata Sockets
Added APIs to expose the metadata endpoints (from api server)
Proxy in worker hosts that listen on sockets and forwards metadata requests to API
Proxy in NSController to forward metadata requests via socket to its worker host.
2026-05-19 02:15:11 +00:00
JamesBhattarai 11c0e000fe Feat(Fix): allow same NS IP across hosts, and L2 Flow isolation
Chore: OVS Flow Code Refractor
2026-05-19 02:15:11 +00:00
JamesBhattarai 35106a2ed7 Feat: GPU VM & GPU-Dynamic Passthrough
API to register GPU (worker does at initialization), list all GPUs (to fetch during creation menu)
Schedular to Match with relevant hosts filter
A Table to store GPU models
 A table to map gpu to pcie to  workload and host
Dynamic binding unbinding of vfio, nvidia driver at runtime for passthrough
2026-05-16 03:59:00 +00:00
JamesBhattarai f4de833cb6 Fix: Redundant add of WorloadHostResources for GPU Discovery Storation per host 2026-05-14 10:12:27 +05:45
coryHawkvelt 28170a2650 Merge pull request 'Feat: Implemented LVM API & Debug Volume Fits Image' (#43) from JamesBhattarai/theapi:v1.01/volumes-xcloudify into master
Reviewed-on: coryHawkvelt/theapi#43
2026-05-13 04:40:29 +00:00
JamesBhattarai 405c5a1093 Feat: Implemented LVM API & Debug Volume Fits Image
API can call volumes.type = lvm
worker_settings.json require  "LVM_VG_NAME"
Check/Debug if image fits into  volume size (local/lvm)
2026-05-06 10:19:59 +05:45
JamesBhattarai c58454c3f5 Feat: DHCP via NSController w OVSFlow & Fix: NSController Delete Schedulars; Fix: NSController Per Network
Implement NS Controller API for Dispatch & Delete to Per Host Per Network
 Chore: Used_ips filter to show deleted (network.py:106) while creating new
DHCP templated w DNSMasq (Mapped Port to IP in NSController)
Protect NSControllers from Delete during Reconsilation
2026-05-05 23:52:42 +05:45
JamesBhattarai fe888efcb8 Feat(Chore): Implement DHCP, NS-Controller IP Store in DB
Also preper ip passed in param when creating port (essentail for NS-Controller) later use.
2026-05-05 23:52:42 +05:45
JamesBhattarai 5e6b8e74f6 Feat: Implemented SSH Key Management 2026-05-05 23:52:42 +05:45
JamesBhattarai 9485e1bcc2 Feat: Add GPU Scheduling + PCIE Discovery Enabled
Warning! : It uses Static, Hardcoded at libvirt_xml_builder.py

Implemented GPUCapableHosts Filter
Santize GPU Payload from User Input
Add Endpoint to register PCIE Devices
Seed the all regions with pcie filter of AMD and NVIDIA GPU
2026-04-22 07:54:37 +05:45
JamesBhattarai bbf71012ea FIX(api): correct WorkloadHostFixedResource field access in _enrich_host_data
Previous code accessed resource.resource_type and resource.quantity which
never existed on WorkloadHostFixedResource. The correct fields are type,
model, manufacturer and physical_address, matching the model definition
and DB schema since the initial migration.

Bug introduced  in 1731e2c.
2026-04-22 07:53:44 +05:45
JamesBhattarai bb357fcbd1 Feat(Fix): VM Networking, Volume,
VM:
Enabled Filter VM Scheduling by  HasSufficientResources
Implemented to Record Resource Usage by WorkloadResourceUsage; creates new record on basis of resource_type (CPU,RAM)

Volume:
Implemented Local Volume vm spin (Fallback Local and reject other volume types for spinning vm),
Fix: hardcode to image for image backed volume in vm payload :298

Network:
Added error handle for port creation, bridge heirarchy selection (network to stored)
2026-04-22 07:53:37 +05:45
JamesBhattarai 3c50a66706 Feat(Compute): Server Groups & Feat(DB): allembic manage.py
Added Affinity/Anti-Affinity Rules for VM & Containers
Allow to see current status, rollback, history of db migrations
2026-04-04 15:53:51 +05:45
JamesBhattarai 99028f40b1 Chore(Feat): added Infra utils
Added Swagger/OpenAPI
Added .env.examples
2026-04-02 19:19:16 +05:45
nexgen_mirrors c7f49cc533 fix(dns): use Flask app config for WebSocket URL
The change switches from using environment variables to Flask's application configuration for retrieving the WebSocket server URL, ensuring consistent configuration access through the application context.
2026-02-04 14:23:26 +10:30
coryHawkvelt a6eb0eb4b7 feat(network): add ovs bridge fallback configuration
Add XCF_DEFAULT_OVS_BRIDGE environment variable to configure a default
OVS bridge for networks without explicit configuration. Update payload
builders to fallback to this default when generating SDN and pod
configurations.

- Add default_ovs_bridge column to regions table via migration
- Update network API routes to support ovs_bridge field
- Implement fallback logic in WorkloadHost and pod payload builders
- Adjust NSController resource allocation (4mCPU/128MiB) and image
2026-01-27 10:42:02 +10:30
coryHawkvelt 62243c4e4b feat(dns): add vdc dns isolation and improve sanitization
Add VDC-specific DNS configuration directories to prevent cross-tenant
DNS leakage. Each VDC now gets its own subdirectory under the global DNS
config path.

Introduce sanitize_dns_name() function to handle edge cases in DNS names
including empty strings, trailing dots, .local suffix duplication, and
consecutive periods.

Add environment variable WEBSOCKET_SERVER_URL for WebSocket server
configuration with fallback to default.

Update dnsmasq configuration to enable expand-hosts and improve DNS
rebind protection.
2026-01-25 06:21:20 +10:30
coryHawkvelt 722b22a5ab chore(api): remove debug logging from get_pods endpoint 2026-01-25 05:39:02 +10:30
coryHawkvelt b00c2a7e00 feat(models): add ovs_bridge field to port configuration output 2026-01-21 18:44:32 +10:30
coryHawkvelt 364e7177fe fix(api): validate network references in container workloads
Add validation to ensure network IDs referenced in container configurations
actually exist and belong to the specified VDC. This prevents processing
containers with invalid or cross-VDC network references, improving data
integrity and providing clear error messages for misconfigured workloads.
2026-01-17 15:57:03 +10:30
nexgen_mirrors ef862078a5 refactor(network): use NSController network port IP for port forwarding
Move NetworkPort creation earlier in the allocation process and use the NSController's attached NetworkPort IP address for port forwarding when available, instead of defaulting to the host's IP address. This ensures pods with NSControllers use their dedicated network port IP for external connectivity.
2026-01-10 19:08:53 +10:30
nexgen_mirrors a152f57539 feat(api): add network ports to pod details response
Collects network ports from all containers within a pod and includes
them in the API response. Each network port entry contains full details
including ID, name, network ID, IP address, MAC address, port type,
DNS servers, subnet mask, and status information.
2026-01-06 15:43:31 +10:30
nexgen_mirrors 52c559e1cd refactor(network): move NetworkPort creation from containers to NSController
NetworkPort objects are now attached to the NSController (not individual containers).
The NSController manages the network namespace for the entire pod, and containers
within that pod share this network configuration. This design allows containers
to share the same network stack while maintaining proper isolation via the
NSController's network namespace.

The refactor consolidates network port creation by:
- Scanning all containers in a pod to collect network requirements
- Creating NetworkPort records attached to the NSController
- Updating port_type from "container" to "nscontroller"
- Removing explicit networks configuration from NSController config (relying on inherited pod networking)
2026-01-06 15:42:42 +10:30
nexgen_mirrors 016f1fc447 refactor(workload): consolidate deletion logic into shared utilities
Extract duplicate deletion code from pod and container routes into
reusable helper functions in deletion_helpers.py module:
- collect_and_cleanup_network_ports() for network port cleanup
- cleanup_workload_related_records() for resource/volume cleanup
- log_deletion_audit_event() for consistent audit logging

Add SDN network port updates during pod deletion to ensure proper
network cleanup when containers are removed.

Optimize get_pods() endpoint by replacing N+1 queries with batch
queries and lookup maps for nscontrollers, tunnels, and DNS records.

Enhance get_pod() response to include cloudflare tunnel configuration
and DNS records for better visibility into pod networking setup.
2026-01-05 16:24:37 +10:30
nexgen_mirrors d1fbd4eaec minor tidy ups 2026-01-05 13:22:00 +10:30
nexgen_mirrors 923dc345ed feat(sdn): add network port data to pod payload
Add network port information to pod payload construction by querying
NetworkPort records for each container and including network details
(vni, ovs_bridge, gateway, ip_address, mac_address, dns_servers) in the
container specification.
2026-01-03 23:04:22 +10:30
nexgen_mirrors 5018a74c3a feat(sdn): add network port management and SDN updates for container lifecycle
Implement comprehensive network port management across container lifecycle operations:
- Add NetworkPort creation when containers are provisioned
- Send SDN updates after port creation for network configuration
- Properly clean up network ports and send SDN updates on container deletion
- Filter out deleted ports from SDN payload calculations and network port queries
- Add dependency on SDN update tasks before container dispatch

These changes ensure proper network state synchronization between the database and SDN controller, preventing stale port configurations and ensuring consistent network topology across container lifecycle operations.
2026-01-02 23:52:04 +10:30
nexgen_mirrors 81dda8de4f fix(sdn): handle None workload_host_ip to prevent VXLAN failures
Added validation to skip peer ports with missing workload_host_ip during SDN
payload generation. When a port's peer host has no east-west IP configured, the
system now logs a warning and continues processing rather than allowing the None
value to cause VXLAN interface failures.
2026-01-02 16:32:57 +10:30
nexgen_mirrors f490f05584 feat(api): add SDN payload preview endpoint for network debugging
Adds a debug endpoint to preview SDN payloads without sending to workers, enabling easier troubleshooting of network configurations. Also fixes indentation issue in generate_sdn_payload that prevented proper return of payload data, and removes blocking debug code that was preventing SDN updates from being processed.
2026-01-02 16:20:12 +10:30
nexgen_mirrors 7b977c7f9b feat(dns): add localhost resolution and DNS configuration for NSControllers
- Add address=/localhost/127.0.0.1 to DNS zone generation for Docker healthchecks
- Configure container DNS to use 127.0.0.1 for NSControllers with empty dns_opt and search
- Update NSController image to xcloudify-nscontroller:latest
- Adjust Dockerfile healthcheck start-period from 10s to 5s for faster startup detection
2026-01-02 14:28:32 +10:30
nexgen_mirrors 1a018edb67 fix(dns): fix SRV record format and add hostNetwork pod support
Correct the SRV record format in dnsmasq configuration from
<service>,<priority>,<weight>,<port>,<target> to
<service>,<target>,<port>,<priority>,<weight>. Also add support for pods
running in hostNetwork mode by using the host's ip_address_northsouth
when a pod has no network port assigned.
2026-01-02 14:12:12 +10:30
nexgen_mirrors 43a0f7ddd0 fix(allocation): fix allocation retry to pick new host on offline failure
When an alternate host doesn't exist for a pod that's failing on an offline
host, we need to clear the workload_host_id reference so that allocate_and_dispatch
will select a new host instead of repeatedly trying the same offline host.
2026-01-02 13:23:32 +10:30
nexgen_mirrors 1a8b3bbaca feat(dns): implement centralized NSController configuration and fix double period bug
Refactor NSController configuration into a single source of truth module, ensuring consistent DNS volume mounts and resource limits. Fixes bug where trailing periods in universe DNS names caused double periods in VDC zone names. Removes legacy DNS update modes and standardizes on shared volume approach. Adds comprehensive configuration constants and improves worker-side DNS handling.
2026-01-02 12:58:03 +10:30
nexgen_mirrors e5e5f8f58f feat(models): add pod relationship to Workload model 2026-01-02 12:56:33 +10:30
nexgen_mirrors 05a9ab1741 dns improvements 2025-12-09 15:16:13 +10:30
coryHawkvelt d0143627fc Anthropics take on the DNS challenge 2025-12-04 10:05:10 +10:30
coryHawkvelt 00e04d69be feat(tasks): schedule downtime handler when marking workers offline
When a worker is marked offline during reconciliation, automatically
schedule the host downtime handler to ensure proper workload
rescheduling. The handler is triggered with a configurable countdown
delay from HOST_DOWNTIME_CONFIRM setting.

This ensures that workloads are properly handled when workers become
unavailable, improving system reliability and task management.
2025-12-04 00:43:46 +10:30
coryHawkvelt 691f845529 feat(api): enhance pod response with detailed port forwarding info
Add external and internal IP addresses, DNS record details, and filter out deleted port forwardings. Include null checks for workload and DNS record.

BREAKING CHANGE: Replaced "ip_address" with "external_ip_address" and added "internal_ip_address" in port forwarding response.
2025-12-03 23:29:50 +10:30