Custom source

Easily plug in your own data integration sources and destinations with Mage Pro.

Mage Pro makes it easy to integrate your own custom data integration sources and destinations.

To get started, follow the adapt an existing source guide or the add a new source guide to create your custom integration.

Once your custom source or destination is ready, simply include it in your Mage Pro project by adding it to the appropriate folder as shown below.

Folder structure

To register custom sources and destinations in your Mage Pro project, organize your project directory as follows:

.
├── mage_data
└── project_name
    ├── charts
    ├── custom
    ├── data_exporters
    ├── data_loaders
    ├── dbt
    ├── extensions
    ├── pipelines
    ├── data_integrations              # ← important
    │   └── sources
    │   │   ├── custom_source_1
    │   │   │   └── templates
    │   │   │   │   └── config.json
    │   │   │   └── __init__.py
    │   │   │   └── README.md
    │   │   ├── custom_source_2
    │   │   └── metadata.yaml          # Register all source definitions here
    │   └── destinations
    │       ├── custom_destination_1
    │       ├── custom_destination_2
    │       └── metadata.yaml          # Register all destination definitions here
    ├── scratchpads
    ├── transformers
    ├── utils
    ├── __init__.py
    ├── io_config.yaml
    ├── metadata.yaml
    └── requirements.txt

Instructions

1. Create a data_integrations folder inside your project directory.
2. Inside data_integrations, create two subfolders:
   • sources: for your custom data source integrations
   • destinations: for your custom data destination integrations
3. Each source/destination should have its own folder (e.g., custom_source_1) containing:
   • An __init__.py file with the integration class
   • A README.md for documentation
   • A templates/ directory with a config.json file for default configuration
4. Define a metadata.yaml file in each of the sources/ and destinations/ folders to register the integrations with Mage Pro.

This structure allows Mage Pro to discover and load your custom integrations dynamically.

Register custom sources and destinations

To register your custom sources or destinations, create or update the metadata.yaml file inside the respective sources/ or destinations/ subfolders under data_integrations.

📘 Example: sources/metadata.yaml

sources:
- name: Source Name
  module_name: SourceModuleName
  uuid: source_uuid

📘 Example: destinations/metadata.yaml

destinations:
- name: Source Name
  module_name: SourceModuleName
  uuid: source_uuid

Field descriptions

• name: Display name shown in the Mage UI dropdown.
• module_name (optional): The class name defined in the source’s __init__.py file. Defaults to the name field with spaces removed.
• uuid (optional): Folder name of the source. Defaults to the lowercased name, with spaces replaced by underscores.

Code modification

Unlike the add a new source guide (which adds sources to the Mage core), using custom sources in your project requires updating import paths to point to your project-level modules.

Update any imports from mage_integrations to data_integrations like so:

# Before
from mage_integrations.sources.custom_source.module import SomeClass

# After
from data_integrations.sources.custom_source.module import SomeClass

This change ensures that Mage loads and uses your custom integration from the project folder instead of the built-in Mage package.

Testing Your Custom Source

To test your custom data source integration:

1. Create a new data integration pipeline.
2. Select your custom source from the source dropdown.
   • Custom sources appear at the bottom of the list in the UI.
3. Enter your source configuration.
4. Click “Test connection” if your custom source class implements the test_connection method.
5. Click “View and select stream” to verify that the expected streams are returned.
6. Add a destination and run the pipeline using a trigger to test end-to-end data flow.

Troubleshooting: Source Not Appearing in UI?

If your custom source isn’t showing up in the Data Integration source dropdown, follow these steps to debug:

1. Create a pipeline and add a scratchpad block.
2. Run the following code in the block to inspect the custom sources:

# Inspect custom source registration
from mage_ai.server.api.integration_sources import get_custom_collection

custom_sources = get_custom_collection('sources', debug=True)
print(f'Custom sources detected: {custom_sources}')

3. Try importing your custom module manually to ensure it’s discoverable and error-free:

from data_integrations.sources.source_uuid import SourceModuleName

Replace source_uuid and SourceModuleName with the actual folder name and class name of your custom source.

Troubleshooting: “View and select stream” Not Working?

If clicking “View and select stream” in the UI doesn’t return any streams, you can manually debug the discovery step using a scratchpad block in your pipeline.

Run the following code to test your custom source’s discover_streams method:

from data_integrations.sources.source_uuid import SourceModuleName

# Replace with your actual config keys and values
config = dict(
    key1='value1',
    key2='value2',
)

# Initialize the source class with config
source = SourceModuleName(config=config)

# Run stream discovery
streams = source.discover_streams()

# Print discovered streams
print(f'Discovered streams: {streams}')

What to check

• Ensure the config you provide contains all required fields expected by your source class.
• If discover_streams() returns an empty list or raises an error, check your source implementation or credentials.