Skip to content

For Developers

This guide helps you set up your development environment and understand the EWE Numera dbt project architecture.

Prerequisites

  • Python 3.11 or higher
  • uv package manager
  • Git with access to the repository
  • Snowflake account with appropriate role grants

Installation

1. Clone the Repository

git clone <repository-url>
cd pattaya

2. Initialize Submodules

The project uses numera-core as a git submodule:

git submodule update --init --recursive

3. Install Python Dependencies

uv sync

4. Configure dbt Profile

Create or update ~/.dbt/profiles.yml:

ewe_numera:
  target: dev
  outputs:
    dev:
      type: snowflake
      account: <your_snowflake_account>
      user: <your_username>
      password: <your_password>  # Or use key-pair auth
      role: NUMERA_EWE_DEV_TRANSFORMER
      warehouse: NUMERA_DEV_TRANSFORMING
      database: NUMERA_EWE_DEV_CORE
      schema: NUMERA_DIM_MODEL
      threads: 4
    prod:
      type: snowflake
      account: <your_snowflake_account>
      user: <service_account>
      role: NUMERA_EWE_PROD_TRANSFORMER
      warehouse: NUMERA_PROD_TRANSFORMING
      database: NUMERA_EWE_PROD_CORE
      schema: NUMERA_DIM_MODEL
      threads: 8

5. Verify Connection

cd dbt
uv run dbt debug

Project Structure

Project Structure

Common Commands

Package Management

# Install/update dbt packages
uv run dbt deps

Running Models

# Run all models
uv run dbt run

# Run specific model with dependencies
uv run dbt run --select +fct_credit_exposure

# Run models by tag
uv run dbt run --select tag:customer

# Run models by folder
uv run dbt run --select models/dimensional/

Testing

# Run all tests
uv run dbt test

# Run tests for specific model
uv run dbt test --select fct_credit_exposure

# Run only singular tests
uv run dbt test --select test_type:singular

Documentation

# Generate documentation
uv run dbt docs generate

# Serve documentation locally
uv run dbt docs serve --port 8080

Debugging

# Verify connection
uv run dbt debug

# Compile SQL without running
uv run dbt compile --select model_name

# View compiled SQL
cat target/compiled/ewe_numera/models/dimensional/fct_credit_exposure.sql

Development Workflow

  1. Create a branch 

    git checkout -b feature/your-feature-name

  2. Make changes to models, tests, or documentation

  3. Test your changes 

    uv run dbt run --select +your_model
uv run dbt test --select your_model

  4. Verify documentation 

    uv run dbt docs generate
uv run dbt docs serve

  5. Commit and push 

    git add .
git commit -m "Description of changes"
git push origin feature/your-feature-name

Configuration Variables

Key variables in dbt_project.yml:

Variable Default Description
dev_mode_limit 1000 Row limit for development queries
ingestion_delay_seconds 0 Delay for incremental processing
metric_start_date 2022-11-01 Earliest metric date to process
volume_precision 10 Decimal precision for volumes
price_precision 6 Decimal precision for prices

Feature Flags

Models can be conditionally enabled via feature flags:

Flag Purpose
feed.clearing_statements Enable clearing bank statement models
repdb_archive Enable E17 archive models
time_series.15m Enable 15-minute power volume models

Schema Organization

Schema Purpose Models
NUMERA_DIM_MODEL Customer-facing dimensional models dim_*, fct_*
ENDUR Endur staging and history models stg_*, sat_*, pit_*
DATA_QUALITY Data quality test results Test failure tables
MONITORING Operational monitoring fct_data_arrival

Data Layers

The project follows a medallion architecture:

  1. Raw (SAT): Full history satellite tables from Endur UDSRs
  2. Staging (STG): Schema application and data cleansing
  3. PIT: Point-in-time snapshots for current state
  4. Mapping: Cross-reference tables linking systems
  5. Dimensional: Customer-facing star schema

See Data Layers for detailed architecture documentation.

numera-core Integration

The project depends on numera-core for:

  • Core dimensional models (dim_deal_attribute, fct_pnl, etc.)
  • Macros for logging, incremental processing
  • Base satellite and staging patterns
  • Party dimension management

EWE-specific models extend or reference numera-core models:

-- Example: Referencing numera-core party dimension
SELECT *
FROM {{ ref('fct_credit_exposure') }} f
LEFT JOIN {{ ref('dim_party_external') }} p
  ON f.dim_party_external_bu_le_attribute_id = p.dim_party_external_bu_le_attribute_id

Troubleshooting

See Troubleshooting for common issues and solutions.

Quick Fixes

Connection failed: 

# Verify Snowflake account format
uv run dbt debug

# Check role grants
SHOW GRANTS TO USER your_username;

Package not found: 

# Clean and reinstall packages
rm -rf dbt_packages/
uv run dbt deps

Model compilation error: 

# View the compiled SQL
uv run dbt compile --select model_name
cat target/compiled/.../model_name.sql

Next Steps