All notable changes to this project will be documented in this file.
- Implemented additional domain categorization rules to handle new gTLDs.
- Improved the 'remove_ns' function to handle a wider range of DNS server domains, including *.NS.CLOUDFLARE.COM.
- Added hashtag functionality for domains, including a new field in the Domain model, an API endpoint for adding hashtags, and UI elements in the list_domains.html template.
- Added timestamps to exported file names.
- Fixed the 'remove_selected' function to correctly return the count of removed domains.
- Corrected the categorization of domains with new gTLDs (e.g., solo.startup is now recognized as a TLD).
- Created separate
models.py
file for database models - Implemented Flask-Migrate for database migrations
- Refactored
app.py
andauth.py
to resolve circular imports - Updated database initialization process
- Resolved circular import issue between
app.py
andauth.py
- Fixed missing
users
table issue by properly initializing the database
- Integrated Flask-Migrate for database migrations
- Created initial database migration for User and Domain models
- Updated User model to use SQLAlchemy ORM
- Modified auth.py to use SQLAlchemy for user operations
- Updated app initialization to include Flask-Migrate
- Resolved issue with missing 'users' table by creating proper database schema
- New
hashtags
field for Domain model - Database migration to add
hashtags
column to the domain table - Flask-Migrate integration for handling database schema changes
- Updated export functionality to include hashtags in exported data
- Modified
list_domains.html
to display hashtags for each domain
- Resolved issue with missing
hashtags
column in domain table
- Display of total domain count in the navigation menu
- Context processor to inject total domain count into all templates
- Sorting functionality for domain list (by domain name and category)
- Extended DNS server removal to include popular providers like Cloudflare and Sedo
- Hashtag functionality for domains
- New field 'hashtags' added to Domain model
- API endpoint for adding hashtags to selected domains
- UI elements in list_domains.html for adding hashtags
- Timestamp added to exported file names
- Updated
list_domains
function to include sorting options - Modified
list_domains.html
template to include sorting links and indicators - Updated
base.html
template to show domain count in the menu - Improved
remove_ns
function to remove a wider range of DNS server domains, including *.NS.CLOUDFLARE.COM - Updated JavaScript in
list_domains.html
to reflect changes in DNS server removal and hashtag functionality - Improved domain categorization to correctly identify new gTLDs like .startup
- Modified export functionality to include timestamps in file names and add hashtags to exported data
- Fixed
remove_selected
function to correctly return the count of removed domains - Corrected categorization of domains with new gTLDs (e.g., solo.startup is now recognized as a TLD)
- Sorting functionality for domain list (by domain name and category)
- Display all domains without pagination
- Updated list_domains route in app.py to support sorting
- Modified list_domains.html template to include sorting links and indicators
- Real-time statistics update after each domain extraction
- Display of current statistics on the index page
- Removed pagination from the domain list page
- Updated domain list filtering to use form submission instead of dynamic updates
- New actions to clean up domain list:
- Remove NS server domains
- Remove subdomains
- Remove selected domains
- Checkboxes for selecting individual domains
- API endpoints for new domain list actions
- Updated list_domains.html to include new action buttons and checkboxes
- Modified app.py to include new API routes for domain list actions
- Dynamic filtering and searching for domain list
- New API endpoint for listing domains with filters
- Updated list_domains.html to use JavaScript for dynamic updates
- Modified app.py to include new API route for listing domains
- Created a base template using Bootstrap for consistent styling
- Updated all HTML templates to extend from the base template
- Improved overall UI/UX with Bootstrap-based design
- Reorganized navigation into a responsive navbar
- New page for listing domains with filtering and pagination
- Search functionality for domains by name and category
- Pagination for the domain list
- Updated navigation to include a link to the new domain list page
- Added flask-paginate to requirements.txt
- Resolved issue with custom rules not being properly loaded in the template
- Database migration script to add email column to users table
- Issue with missing email column in users table
- Implemented caching mechanism using Flask-Caching
- Added caching to search, export, API, and statistics routes
- Updated app.py to include caching configuration and decorators
- Added Flask-Caching to requirements.txt
- User registration functionality
- New registration page and form
- Email field for user accounts
- Updated User model to include email field
- Modified auth.py to support user creation
- Updated login page to include link to registration
- Support for custom domain categorization rules
- New page for managing custom rules
- JSON file storage for custom rules
- Updated domain_extractor.py to use custom rules in categorization
- Modified app.py to include routes for managing custom rules
- Updated index.html to include a link to the custom rules page
- Implemented automated testing with GitHub Actions
- Created GitHub Actions workflow for running tests
- Expanded unit tests in test_app.py
- Updated TODO.md to reflect the implementation of automated testing
- Implemented multi-factor authentication using TOTP
- New routes for two-factor authentication and enabling 2FA
- Added pyotp library for generating and verifying TOTP tokens
- Updated User model to include two_factor_secret
- Modified login process to support 2FA
- Added new templates for 2FA authentication and setup
- Email notifications for bulk import results
- Flask-Mail integration for sending emails
- Updated bulk_import function in app.py to send email notifications
- Added Flask-Mail to requirements.txt
- Implemented data visualization for domain statistics
- New statistics page with pie chart showing domain categories
- Added Chart.js library for creating interactive charts
- Updated app.py to include a new route for domain statistics
- Modified index.html to include a link to the statistics page
- Support for exporting domains to Excel format
- Added openpyxl library for Excel file generation
- Updated export functionality in app.py to include Excel export
- Modified index.html template to include Excel export option
- Updated requirements.txt with openpyxl dependency
- Improved documentation in README.md
- Added usage instructions for CLI in README.md
- Updated init.sh script to include CLI setup
- Minor bug fixes and code improvements
- Command-line interface (CLI) for the application
- New CLI commands: extract, validate, and list_domains
- Click library for building the CLI
- Updated requirements.txt to include Click library
- Improved documentation in README.md
- Created git_update.sh script for easier version management
- Updated init.sh script to create git_update.sh
- Refactored domain_extractor.py to use tldextract for better domain categorization
- Updated app.py to include admin functionality and rate limiting
- Resolved compatibility issues with markdown package
- Updated markdown package to version 3.4.1 to resolve compatibility issues
- Modified init.sh script to force reinstall packages
- Updated init.sh script to use python3 explicitly
- Added importlib-metadata to requirements.txt to resolve compatibility issues
- Implemented more sophisticated domain categorization using tldextract
- Added user roles (admin and regular user)
- Implemented rate limiting for API access
- Created an admin page for user management
- Updated domain_extractor.py to provide more detailed categorization
- Modified auth.py to include user roles
- Updated app.py to include rate limiting and admin functionality
- User authentication and authorization
- Bulk domain import from CSV files
- Domain validation and categorization
- RESTful API for programmatic access
- Improved error handling and logging
- Search functionality for domains
- Export options for CSV and JSON formats
- Responsive design for the web interface
- Updated Flask and Werkzeug versions to 2.0.3 for compatibility
- Restructured the application to use separate modules for authentication and domain extraction
- Duplicate domain removal in extraction process
- Initial release of the Domain Extractor application
- Basic domain extraction from plain text
- Simple web interface for input and display of results
- Display of total domain count in the navigation menu
- Context processor to inject total domain count into all templates
- Sorting functionality for domain list (by domain name and category)
- Extended DNS server removal to include popular providers like Cloudflare and Sedo
- Hashtag functionality for domains
- New field 'hashtags' added to Domain model
- API endpoint for adding hashtags to selected domains
- UI elements in list_domains.html for adding hashtags
- Timestamp added to exported file names
- Updated
list_domains
function to include sorting options - Modified
list_domains.html
template to include sorting links and indicators - Updated
base.html
template to show domain count in the menu - Improved
remove_ns
function to remove a wider range of DNS server domains, including *.NS.CLOUDFLARE.COM - Updated JavaScript in
list_domains.html
to reflect changes in DNS server removal and hashtag functionality - Improved domain categorization to correctly identify new gTLDs like .startup
- Modified export functionality to include timestamps in file names and add hashtags to exported data
- Fixed
remove_selected
function to correctly return the count of removed domains - Corrected categorization of domains with new gTLDs (e.g., solo.startup is now recognized as a TLD)