Python noob help

cartman82

I'm doing a little side-project in python (flask + pymongo). Just for fun, and as an excuse to learn the language.

As usual, I'm down with basic programming, but I'm struggling with what could be considered "best practices" in structuring the code.

Here's where I'm stuck:

In my main module, I'm doing something like this:

(app/__init.py)

app = Flask(__name__, instance_relative_config=True)
app.config.from_object('default_config')
app.config.from_pyfile('config.py', silent=True)

Included in this config are the settings for my mongo db. In a different module I am thinking to put my business logic / data access methods.

(app/data/users.py)

from . import db

def list_all_users():
    return db.users.find()

So all I need now is to initialize the shared db object in the module's __init__.py. A naive solution would be something like this:

(app/data/__init.py)

from app import app
config = app.config['MONGODB']
client = MongoClient(config['host'], config['port'])
db = client['cartman_db']

This is, of course, terrible. Why would a db module know about my app module? It should just be given (you might even say injected) the info it needs.

However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.

So I tried doing this:

(app/__init.py)

# other init stuff...
import app.data as data
data.connect(**app.config['MONGODB'])

(app/data/__init.py)

client = None
db = None

def connect(host='localhost', port=27017):
    global client, db
    client = MongoClient(host, port)
    db = client['cartman_db']

But that, of course, screws with my import strategy in app/data/users.py (it gets None values initially and doesn't update).

So now, I've started looking into singletons or wrapping these globals into dictionaries, or having a "getter" function... but honestly, this all seems hacky as hell.

Am I even on the right track? I can probably make this work, but I don't think I can make myself like it.

What is the proper way to do this in python?

accalia

I'm no expert on python best practices, but when i have that sort of shared resource problem i do tend to reach for a singleton pattern.

have a method produce a singleton on demand for your shared resource and then just include the static method in a utils or other class and call it when you want the resource. if it's already been constructed it hands it to you, or creates one and saves it for next time too if it hasn't already been used.

it might be a bit overkill and possibly not the best pattern but it's one i know of off the top of my head and it gets the job done correctly with little additional thought on my part.

cartman82

Yeah, that would be something like this:

(app/data/__init.py)

_client = None
_db = None

def connect(host='localhost', port=27017):
	global _client, _db
	_client = MongoClient(host, port)
	_db = _client['cartman_db']

def db():
	if not _db:
		raise Exception('Not connected')
	return _db

Then, from DAL methods:

(app/data/users.py)

from . import db

def list_all_users():
    return db().users.find()

Or do it with a real singleton object. Still seems hacky.

accalia

it is a bit.

gets the job done though.

tar

I am not a python, but I would probably make a class that takes a config file in its constructor, and has a map/dictionary/hash-like interface. Maybe even go as far as creating an IConfigurable and having your config class implement it. Then any other classes can take an IConfigurable in their constructors, and they can query it for their configuration state.
Here, some vaguely C#-ish pseudocode—it should be easy enough to translate into snake language:



interface IConfigurable {

IConfigurable subConfig(string name);

string operator[](string name);

}

Then, in the startup file:



main() {

IConfigurable config = new IniConfig("app.ini");

Application myApp = new Application(config);

while(myApp.update()) {} // or whatever the main loop looks like...

//...

}

later in the application:



Application::Application(IConfiguration config) {

m_db = new Database(config.subConfig("database"));

m_socket = new Socket(config.subConfig("socket"));

//...

}

cartman82

Not really KISS, since my flask app is already loading and parsing all the configuration. I would do it similar to what you're suggesting in C*, but python seems to have a different philosophy. I'm sort of trying to "go native" here as much as possible.

Ok, here's where I'm at now.

(app/data/__init__.py)

from pymongo import MongoClient


class _DataContext(object):
	_client = None
	_db = None

	def connect(self, host='localhost', port=27017):
		self._client = MongoClient(host, port)
		self._db = self._client['cartman_db']

	@property
	def db(self):
		if not self._db:
			raise Exception('Not connected')
		return self._db

	def __getattr__(self, item):
		return self.db[item]


dc = _DataContext()

So, _DataContext is a private class that I'm using as a singleton. It in itself is not a Singleton, ensuring I can mess with it in the tests I will probably never write.

__getattr__ is an extra touch enabling me to use it like this:

(app/__init__.py)

from app.data import dc
dc.connect(**app.config['MONGODB'])

(app/data/users.py)

from . import dc

def list_all_users():
    return dc.users.find()

This feels "right" in terms of interface. Customizable enough and elegant to use.

Thoughts? Hidden gotchas? Performance concerns?

tar

I guess my core point was: don't use Singletons, always pass a configuration object into each component's ctor. (Having said that, if your config is unlikely to change throughout the app's lifetime, you can probably get away with using a singleton even though it's morally wrong.)

You already have dc. as the root of all config lookups, so dc may as well be local to the ctor, or a member variable on the class, rather than being some mysterious object that appears in scope via chicanery.

Python makes you pass self into methods, doesn't it? So I could argue that passing in the dc is Pythonic :<vvx>)

Filed under: $0.02

cartman82

@tar said:

You already have dc. as the root of all config lookups, so dc may as well be local to the ctor, or a member variable on the class, rather than being some mysterious object that appears in scope via chicanery.

I'm still learning, but I'm pretty sure that's not how python works. There are no "super-globals", all globals are limited to their module. So things never appear mysteriously, you always have to explicitly import them.

from app.data import dc

See? Also, it seems that modules are already sort of singletons in a sense that python runtime instantiates them as singletons and that's what you get from import calls. So you don't really need a singleton class as such. A common practice (it seems from SO) is to just create an instance inside the module's __init__ and use import statements as a singleton getter, to pass the instance around. Which is what I'm doing above.

I think you're sort of projecting your C* "instincts" and patterns into a very different language. That said, you might also be right that I should just find a DI container instead of patching things together with native modules.

However, baring a slap from a 'native' python guru, I think I'll keep the pattern I have now.

tar

Fair enough. I'm probably being "the opposite of useful".

cartman82

@tar said:

Fair enough. I'm probably being "the opposite of useful".

No prob, it helps me learn when I'm explaining things. Blind leading the blind won't get far, but at least they are on a journey.

lucas

Why are your prefixing stuff with an underscore?

cartman82

Python doesn't have real private / protected syntax, so that is a convention. A single underscore "_" in modules and "__" in classes.

There are also language constructs that will act differently depending on the underscoring. For example, if I do:

from app.data import *

...I will get dc variable, but not _DataContext class. I can still access it, though, like this:

import app.data as data

dc = data._DataContext()

lucas

Cheers I am a total noob with python and been writing a few scripts (boring data import stuff), though the stuff I happened to have written are scripts.

dkf

@cartman82 said:

Python doesn't have real private / protected syntax, so that is a convention. A single underscore "_" in modules and "__" in classes.

I believe the leading _ triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.

Eldelshell

Don't put code in init.py files. Package level values yes, logic code no, and heavy (DB) code never. There are some articles explaining why but basically, it's not the correct way since this files are only to mark a directory as a package, not for code.

Now, there's Flask-PyMongo which maps & connects your models to a MongoDB. Use it, because AFAIR it also provides caching and seamless conversion.

cartman82

@dkf said:

I believe the leading _ triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.

That's right.

@Eldelshell said:

Don't put code in init.py files. Package level values yes, logic code no, and heavy (DB) code never. There are some articles explaining why but basically, it's not the correct way since this files are only to mark a directory as a package, not for code.

Got it. I'll move the _DataContext class to a different file.

@Eldelshell said:

Now, there's Flask-PyMongo which maps & connects your models to a MongoDB. Use it, because AFAIR it also provides caching and seamless conversion.

I hate it. As far as I can see, all it does is fuse your db directly to your API in the worst thick-client manner possible.

From their site:

Collection.find_one_or_404(*args, **kwargs)
Find and return a single document, or raise a 404 Not Found exception if no document matches the query spec. See find_one() for details

Ugh. Also, all their examples happily dig directly into the database from the view methods. Call me old-fashioned but I need at least one layer of separation there for the peace of mind.

Also, the docs don't say anything about models or caching. Just show you a bunch of terrible shortcuts you should never take.

dkf

@cartman82 said:

Collection.find_one_or_404

WAT?

PleegWat

@dkf said:

WAT?

What he said.

Eldelshell

That's what you get for using MongoDB :-p

asdf

@Eldelshell said:

That's what you get for using MongoDB

At least his project is web scale.

Bulb

@accalia said:

singleton pattern

I've let¹ them in our project at $work couple of years ago, before I read² Singletons: Solving problems you didn’t know you never had since 1995³. I really regret it.

Edit: Weird clipboard behaviour in chromimum strikes again. Actually did what I said in ³.

accalia

jalf.dk » Blog Archive » Singletons: Solving problems you didn’t know you never had since 1995

added wayback machine link because source site is offline

/me begins to read the article

huh... well this i will need to think on....

yes think carefully.

Bulb

@cartman82 said:

However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.

Design patterns are not language-specific. Their realization might be—interface in python is just a documented method with specific arguments, many things can be simple functions in python where you'd need a class and an interface in Java and such—but the concepts remain.

I do sometimes have feeling that especially Java community has many Architecture Astronauts who overuse the patterns even in cases where they don't really bring anything and just make the code longer though. The python approach is to follow YAGNI: don't add more words until it actually gives you some benefit.

Bulb

@accalia said:

added wayback machine

I opened wayback machine when I was writing it and then chromium didn't select the link as I expected and I didn't notice it pasted the wrong thing. Lately chromium (might be X, but I don't recall this problem when selecting from terminal) only sets the active selection when I select something the second time. It is clearly intended to avoid wiping it with the auto-select when clicking into entry so it can be pasted over, but it ends up behaving weird.

tarunik

@cartman82 said:

However, Stack Overflow says dependency injection in python world isn't the darling it is in C#/Java land and seems like overkill for something this simple.

It's more that dependency injection frameworks aren't a darling of the Python world -- manual DI is alive and well in Python, we just don't build entire programs around DI frameworks because it makes much less sense to do so when your programming language isn't [s]a pile of verbose spew[/s]Java.

@cartman82 said:

See? Also, it seems that modules are already sort of singletons in a sense that python runtime instantiates them as singletons and that's what you get from import calls. So you don't really need a singleton class as such. A common practice (it seems from SO) is to just create an instance inside the module's init and use import statements as a singleton getter, to pass the instance around. Which is what I'm doing above.

QFT -- Python modules make better singletons than most instances of the Singleton pattern. Not that Singletons are all that they're chalked up to be...

@dkf said:

I believe the leading _ triggers a rewriting of the real method name to something else (that includes a munged class name IIRC?) so giving pseudo-privacy.

It's the leading __, and only inside classes. In Python, _ is used for internal methods/functions (rarely, internal variables as well), and __ is used for 'true' privacy because of the munging scheme.

cartman82

@tarunik said:

It's more that dependency injection frameworks aren't a darling of the Python world -- manual DI is alive and well in Python, we just don't build entire programs around DI frameworks because it makes much less sense to do so when your programming language isn't a pile of verbose spewJava.

Yeah I think I'll just keep using native modules. Python has a good system, reminds me of common.js a bit (if you avoid using "from module import *" crap).

As for the Singleton, the problem isn't that you want only one instance of a class in your app (a legit requirement), but that you want one instance in ANY app EVER. That's pushing a concern of a specific app down into a class. A wrong approach.

dkf

@cartman82 said:

As for the Singleton, the problem isn't that you want only one instance of a class in your app (a legit requirement), but that you want one instance in ANY app EVER. That's pushing a concern of a specific app down into a class. A wrong approach.

Exactly. Singleton by configuration is fine.

boomzilla

@Bulb said:

Lately chromium (might be X, but I don't recall this problem when selecting from terminal) only sets the active selection when I select something the second time.

Hmm....I haven't noticed that, exactly, but I've learned not to rely on active selection on chrome / chromium. Even other clipboard actions are a bit wonky. When going between VMs I find myself pasting into Kate and copying from there before it will make the jump.